From Hypertopic

Fragment entities load time

Problem description

Loading a Cassandre viewpoint referencing 7000 fragments with Porphyry took 3:20.

Note : The solution must work for textual AND image fragments in Porphyry.

Solution 1: Do not load fragments (Porphyry)

Comment by Christophe

This solution would undoubtedly speed up the loading. However, if the source payload is kept unchanged, Porphyry will display references of fragment (like /source/0+9 ) instead of fragment preview. I do not known if this is a problem in a image context (given that source panel enable to quickly recognize the image), but I am sure that, in a textual context, such a fragment is meaningless. Anyway, perhaps some users would prefer speed to preview. But, some user won't use Porphyry anymore if they are not able to browse all the fragment at one glance (which is one of the perceived core advantage compared to Agorae). Moreover, such an appearance would not be consistent with the qualitative epistemology of the tool. I however do not oppose to this solution, but it is subject to raise other problems.

Solution 2: PCDATA in fragment list (source payload)

For a text fragment (already implemented in Argos):

<entity>
  <relatedEntity relationType="includes" href="1+7057">This very short text contains only one sentence</relatedEntity>
  <attribute name="type" value="source"/>
</entity>

For an image fragment:

<entity>
  <relatedEntity relationType="includes" href="100+800+400+1200">http://acme.org/getFragment?file=foo&coord=100+800+400+1200&max_height=100</relatedEntity>
  <attribute name="type" value="source"/>
</entity>

Comment by Aurélien

⊗ The PCDATA has to be parsed to know if it is a url or a text.

⊗ Wikimedia fail shows that the ampersand will be interpreted as an XML entity!

Comment by Christophe

In order to avoid the interpretation, URL could be encoded (?)

Solution 3: xhtml element in fragment list (source payload)

For a text fragment (already implemented in Argos):

<entity>
  <relatedEntity relationType="includes" href="1+7057">This very short text contains only one sentence</relatedEntity>
  <attribute name="type" value="source"/>
</entity>

For an image fragment:

<entity>
  <relatedEntity relationType="includes" href="100+800+400+1200">
    <html:img src="http://acme.org/getFragment?file=foo&coord=100+800+400+1200&max_height=100"/>
  </relatedEntity>
  <attribute name="type" value="source"/>
</entity>

Comment by Aurélien

⊕ Images and text can be inserted in the same way in a web page or in a Swing label. Easy to get with XSLT, Xpath, DOM.

⊗ Hard to get wih SAX (which is used by Porphyry). Or is it possible to genererate events only on one namespace?

Comment by Christophe

The Steatite strategy could apply for both Image and Text (Steatite-produced fragments appear to be more efficient than SQL requests).

<entity>
  <relatedEntity relationType="includes" href="1+7057">
    <html:a href="http://acme.org/getFragment?file=foo&coord=1+7057"/>
  </relatedEntity>
  <attribute name="type" value="source"/>
</entity>

As an advantage, this would be very fast (and light) for the server. As a disadvantage, it will result in as many requests (from the client) as featured fragments. [Thinking] Numerous requests would take longer than previous xhtml (or similar PCDATA) solution. [Thinking twice] I realize that this option is almost the same as Solution 5.

Solution 4: work offline

The idea is create a local copy of hypertopic service. If the user want to visit a hypertopic service, he should download a copy of hypertopic database (all hypertopic objects and fragments). Then the client (porphyry) should continuously synchronize your local database. For example:

1. I want to use hypertopic service http://argos.hypertopic.org in porphyry, I open this service in porphyry. Porphyry will down load all the data from http://argos.hypertopic.org. 2. Porphyry will start a service to synchronize the local copy by subscribing the trace/operations RSS from argos.hypertopic.org.

Comment by Christophe

• Perhaps this offline solution is similar to cache optimisation. The synchronization tasks can be achieved with code 304.
• Moreover, this solution can be combined with previous ones. For instance : PCDATA & Offline.

Comment by Aurélien

We are not researchers in middleware so i am afraid we cannot spend time on this solution. Moreover, if we propose a work-around and do not consider this problem as a refutation of our current protocol, this means that our scientific work is not refutable...

Comment by Christophe

I am very sensitive to this comment. I agree.

Solution 5: Referencing steatite in fragment list (source payload)

For a text fragment:

<entity>
  <relatedEntity relationType="includes" href="1+7057" preview="http://acme.org/Text/getFragment?file=foo&coord=1+7057"/>
  <attribute name="type" value="source"/>
</entity>

Comment by Christophe

I did envisage something close to that previously. This indeed is a solution (but, we have to admit that, it is less straightforward that the PCDATA/xhtml option)

For an image fragment:

<entity>
  <relatedEntity relationType="includes" href="100+800+400+1200" preview="http://acme.org/Image/getFragment?file=foo&coord=100+800+400+1200&max_height=100"/>
  <attribute name="type" value="source"/>
</entity>

In short

• Giving fragment details directly in topics provides easy display in Agorae: texts should be inline, images should be given as a URL (XSLT conversion from Hypertopic to HTML).

• Using an html:img element for image reference would make Xpath processing slightly easier but SAX processing (as in Porphyry) really more difficult. We should use an attribute instead.

• Giving fragment details directly in sources can be an interesting feature. However it does not provide a benefit in loading time since Porphyry will load topics too.

• Steatite text fragment extraction seems to be faster than Cassandre SQL aggregation. However to be inserted inline in Cassandre, Steatite getFragment must be rewrote in order to return plain text instead of HTML.

• Note: By changing the extract coordinates, it may be possible to have a KWIC like feature (that would make loading different extracts from the same fragment useful).

Comment by Christophe

3: If Porphyry loads topics, then load sources containing the fragments refered in topics (avoiding loading separately fragments themselves), then it will provide a benefit for loading widely topiced corpus (while projects with few topics will take longer to load).

4: SQL and Steatite strategies are currently under comparision...

5: Christophe doesn't see how to proceed yet; but he is really interested in such a KWIC feature ;-)

How to remove a ressource ?

I'm a student and I tried to remove a ressource from an entity. I use the PUT method (to update an entity) but I get an 500 Error.

The PUT method is working if you put new ressources or changes the ressources but, when I tried to remove a ressource I get an SQL error like "Put entity error:SQLSTATE[42703]: Undefined column: 7 ERROR: column "resoucename" does not exist LINE 1: ...sources where serviceurl=$1 AND entitypath=$2 AND resoucenam...".

I'm wondering how can I delete a ressource from an entity. Do I need to delete and create the entity again with new ressources ? Can I use a POST method ?

COMMENT BY Benjamin Hilaire (SIT4 - IL)