Parsing large XML responses

[zemanel]

I'm currently not able to parse large endpoint responses due to memory issues (one of the endpoints is a list with 22,746 value entries and no result pagination).

A solution is apparently (as far as i know at the moment) parsing the response using stream parser like SAX (http://developer.android.com/reference/javax/xml/parsers/SAXParser.html). Taking a look at the library, it would be either refactoring or creating a paralel implementation of the ResponseParser and data type Serializer interfaces and implementations (since they are tied to org.w3c.dom.Element).

What do you think?

[timroes]

I think SAX parser would to be preferred in general, since you are reading all the time from network. So I would replace the DOM parser through a SAX parser, if someone (hint hint) would write it :)
If you don't want to give it a look, I would give it a look within the next days. Just give me a hint, if you want to look at it, that we don't do the same work twice.

[zemanel]

I've been reading documentation regarding using the SAX parser on Android [compatibility] but i'm unsure on how to refactor the library at the moment, if the current implementation should be replaced or a SAX based parsed added to it for example.

[timroes]

I will change it so SAX parser tomorrow (German time). So if you have around 24 hours time from now, you will get a working SAX version. As it seem (https://github.com/zemanel/pypi-java-xmlrpc-client/tree/develop#implemented-xmlrpc-calls) you still have some other methods, you can implement first :-)

[zemanel]

No rush and i would be glad to help :-) yes actually i was going over that; currently implementing the "search" method call of:

http://wiki.python.org/moin/PyPiXmlRpc

p.s.: the library you mentioned is a component of another project, https://github.com/zemanel/pypi-android , which will i hope will be a good app for tracking Python package releases, someday :-)

[timroes]

For documentation reasons: I profiled your project, and as it seems most of the objects allocations will be there, even with SAX parser. But I will keep that in mind for later, and might give it a try some time, but not now, as I see no large chance, it will help the problem.