Do you have an update for me about your recent findings yet?
Meanwhile I have created an additional Unit test to better understand how this issues could occur and what possible improvements option are.
Here is the test configuration:
/**
* The tested columns.
*/
private static final int[] TEST_EXTREMELY_SPARSE_TABLE_COLUMS = { 1, 2, 3, 4, 5, 6 };
/**
* These are the requests that are send on behalf of TableUtils when doing a table walk for
* {@link #TEST_EXTREMELY_SPARSE_TABLE_RESPONSE_PDUS} simulated table data. For better readability, only the
* suffixes of the OIDs are included. The required prefixes are added programmatically in the test.
*/
private static final int[][][] TEST_EXTREMELY_SPARSE_TABLE_REQUEST_PDUS =
{
{ { 1 }, { 2 }, { 3 }, { 4 }, { 5 }, { 6 } },
{ { 1,103 }, { 2,104 }, { 3,103 }, { 5,104 }, { 6,103 } },
{ { 1,105 }, { 2,106 }, { 3,105 } },
};
/**
* This table contains a GETBULK response PDU per first array dimension. In the second, each
* {@link VariableBinding}'s {@link OID} is given which is returned by the simulated agent to
* that tested {@link TableUtils}. Again, only the suffixes of the OIDs are included here.
*/
private static final int[][][] TEST_EXTREMELY_SPARSE_TABLE_RESPONSE_PDUS =
{
{ { 1,100 }, { 2,100 }, { 3,100 }, { 5,100 }, { 5,100 }, { 6,101 } ,
{ 1,102 }, { 2,103 }, { 3,102 }, { 5,101 }, { 5,101 }, { 6,102 } ,
{ 1,103 }, { 2,104 }, { 3,103 }, { 5,104 }, { 5,104 }, { 6,103 } },
{ { 1,104 }, { 2,105 }, { 3,104 }, /*{ 5,105 },*/ { 5,106 }, { 6,104 },
{ 1,105 }, { 2,106 }, { 3,105 }, /*{ 5,106 },*/ { 6,100 }, { 8,105 },
{ 2,107 }, { 3,100 }, { 4,102 }, /*{ 6,100 },*/ { 7,101 } },
{ { 2,107 }, { 3,105 }, { 4,104 },
{ 2,108 }, { 3,106 }, { 4,105 },
{ 2,109 }, { 3,107 }, { 4,106 } },
};
/**
* The {@link #TEST_EXTREMELY_SPARSE_TABLE_EXPECTED_ROWS} array contains the {@link TableEvent} row structure
* expected to be returned by the tested {@link TableUtils}.
*/
private static final int[][][] TEST_EXTREMELY_SPARSE_TABLE_EXPECTED_ROWS = {
{ { 1,100 }, { 2,100 }, { 3,100 }, null, { 5,100 }, null },
{ null, null, null, null, { 5,101 }, { 6,101 } },
{ { 1,102 }, null, { 3,102 }, null, null, { 6,102 } },
{ { 1,103 }, { 2,103}, { 3,103 }, null, null, { 6,103 } },
{ { 1,104 }, { 2,104}, { 3,104 }, null, { 5,104 }, { 6,104 } },
{ { 1,105 }, { 2,105 }, { 3,105 }, null, null, null },
{ null, { 2,106 }, null, null, { 5,106 }, null },
};
The test succeeds and shows, that TableUtils is optimising pretty well if columns are early finished. Like column 4 after the first request and then 5 and 6 after the second.
There are a couple of further improvements possible but all of them further increase the complexity.
For your issue with the large row cache, mostly relevant is the fact, that if the last column of the first row in the cache is not returned by the agent (like in this test case), the row cache grows until the last row is received.
The above behaviour can be improved as follows:
- Whenever a column is recognised to be finished,
TableUtils must check if TableEvents in the cache can be released to the listener because that row can be now implicitly taken as finished.
- A global row cache limit could be implemented, that ensures row
TableEvent release if there are more >2 times (configurable) more rows in the cache than maxRepetitions for the GETBULK configured.
The second optimisation configuration might cause sparse rows in large tables dynamically created during retrieval to be returned incomplete, when column retrieval is distributed over several PDUs. But this is really an edge case and can be mitigated by the denseTableDoubleCheckIncompleteRows option.
I am going to implement the first proposed optimisation first…
