Package org.apache.daffodil.api
Overview
TheDaffodil object is a factory object to create a
Compiler. The Compiler provides
a method to compile a provided DFDL schema into a ProcessorFactory,
which creates a DataProcessor:
Compiler c = Daffodil.compiler();
ProcessorFactory pf = c.compileFile(file);
DataProcessor dp = pf.onPath("/");
The DataProcessor provides the necessary functions to parse and
unparse data, returning a ParseResult or
UnparseResult, respectively. These contain information about the
parse/unparse, such as whether or not the processing succeeded with any diagnostic information.
Compiling a DFDL schema is expensive and the compiled objects are large, so they are designed to be created once and shared. See Thread Safety, at the end of this document, for which of these objects may be shared by multiple threads and which must be unique to each thread.
TheDataProcessor also provides two functions that can be used to
perform parsing/unparsing via the SAX API. The first creates a
DaffodilParseXMLReader which is used for parsing, and the
second creates a DaffodilUnparseContentHandler which is used for
unparsing.
DaffodilParseXMLReader xmlReader = dp.newXMLReaderInstance();
DaffodilUnparseContentHandler unparseContentHandler = dp.newContentHandlerInstance(output);
The DaffodilParseXMLReader has several methods that allow one to
set properties and handlers (such as ContentHandlers or ErrorHandlers) for the reader. One can
use any contentHandler/errorHandler as long as they extend the
ContentHandler and ErrorHandler interfaces
respectively. One can also set properties for the DaffodilParseXMLReader
using DaffodilParseXMLReader.setProperty(java.lang.String, java.lang.Object).
The following properties can be set as follows:
The constants below have literal values starting with "urn:ogf:dfdl:2013:imp:daffodil.apache.org:2018:sax:" and ending with "BlobDirectory", "BlobPrefix" and "BlobSuffix" respectively.
xmlReader.setProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_BLOBDIRECTORY(),
Paths.get(System.getProperty("java.io.tmpdir"))); // value type: java.nio.file.Paths
xmlReader.setProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_BLOBPREFIX(), "daffodil-sax-"); // value type String
xmlReader.setProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_BLOBSUFFIX(), ".bin"); // value type String
The properties can be retrieved using the same variables with
DaffodilParseXMLReader.getProperty(java.lang.String) and casting
to the appropriate type as listed above.
The following handlers can be set as follows:
xmlReader.setContentHandler(contentHandler);
xmlReader.setErrorHandler(errorHandler);
The handlers above must implement the following interfaces respectively:
org.xml.sax.ContentHandler
org.xml.sax.ErrorHandler
The ParseResult can be found as a property within the
DaffodilParseXMLReader using this uri:
"urn:ogf:dfdl:2013:imp:daffodil.apache.org:2018:sax:ParseResult" or
DaffodilParseXMLReader.DAFFODIL_SAX_URN_PARSERESULT.
In order for a successful unparse to happen, the SAX API requires the
unparse to be kicked off by a parse call to any XMLReader implementation that
has the DaffodilUnparseContentHandler registered as its content
handler. To retrieve the UnparseResult, one can use
DaffodilUnparseContentHandler.getUnparseResult() once the
XMLReader.parse run is complete.
Parse
DataProcessor Parse
TheDataProcessor.parse(org.apache.daffodil.api.InputSourceDataInputStream, org.apache.daffodil.api.infoset.InfosetOutputter) method accepts input data to parse in the form
of a InputSourceDataInputStream and an
InfosetOutputter to determine the output representation
of the infoset (e.g. Scala XML Nodes, JDOM2 Documents, etc.):
JDOMInfosetOutputter jdomOutputter= new JDOMInfosetOutputter();
InputSourceDataInputStream is = new InputSourceDataInputStream(data);
ParseResult pr = dp.parse(is, jdomOutputter);
Document doc = jdomOutputter.getResult();
The DataProcessor.parse(org.apache.daffodil.api.InputSourceDataInputStream, org.apache.daffodil.api.infoset.InfosetOutputter) method is thread-safe and may be called multiple
times without the need to create other data processors. However,
InfosetOutputter's are not thread safe, requiring a
unique instance per thread. An InfosetOutputter should
call InfosetOutputter.reset() before reuse (or a new one
should be allocated). For example:
JDOMInfosetOutputter jdomOutputter = new JDOMInfosetOutputter();
for (File f : inputFiles) {
jdomOutputter.reset();
InputSourceDataInputStream is = new InputSourceDataInputStream(new FileInputStream(f)));
ParseResult pr = dp.parse(is, jdomOutputter);
Document doc = jdomOutputter.getResult();
}
One can repeat calls to parse() using the same InputSourceDataInputStream to continue parsing
where the previous parse ended. For example:
InputSourceDataInputStream is = new InputSourceDataInputStream(dataStream);
JDOMInfosetOutputter jdomOutputter = new JDOMInfosetOutputter();
boolean keepParsing = true;
while (keepParsing && is.hasData()) {
jdomOutputter.reset();
ParseResult pr = dp.parse(is, jdomOutputter);
...
keepParsing = !pr.isError();
}
SAX Parse
TheDaffodilParseXMLReader.parse(org.apache.daffodil.api.InputSourceDataInputStream) method accepts input data to parse in
the form of a InputSourceDataInputStream. The output
representation of the infoset, as well as how parse errors are handled, are dependent on the
content handler and the error handler provided to the DaffodilParseXMLReader.
For example the SAXHandler provides a JDOM representation, whereas
other ContentHandlers may output directly to a OutputStream or Writer.
SAXHandler contentHandler = new SAXHandler();
xmlReader.setContentHandler(contentHandler);
InputSourceDataInputStream is = new InputSourceDataInputStream(data);
xmlReader.parse(is);
ParseResult pr = (ParseResult) xmlReader.getProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_PARSERESULT());
Document doc = saxHandler.getDocument();
The The DaffodilParseXMLReader.parse(org.apache.daffodil.api.InputSourceDataInputStream) method is not thread-safe and may
only be called again/reused once a parse operation is completed. This can be done multiple
times without the need to create new DaffodilParseXMLReaders, ContentHandlers or ErrorHandlers.
It might be necessary to reset whatever ContentHandler is used (or allocate a new one). A
thread-safe implementation would require unique instances of the DaffodilParseXMLReader and its
components. For example:
SAXHandler contentHandler = new SAXHandler();
xmlReader.setContentHandler(contentHandler);
for (File f : inputFiles) {
contentHandler.reset();
InputSourceDataInputStream is = new InputSourceDataInputStream(new FileInputStream(f));
xmlReader.parse(is);
ParseResult pr = (ParseResult) xmlReader.getProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_PARSERESULT());
Document doc = saxHandler.getDocument();
}
The value of the supported features cannot be changed during a parse, and the parse will run
with the value of the features as they were when the parse was kicked off. To run a parse with
different feature values, one must wait until the running parse finishes, set the feature values
using the XMLReader's setFeature and run the parse again.
One can repeat calls to parse() using the same InputSourceDataInputStream to continue parsing
where the previous parse ended. For example:
InputSourceDataInputStream is = new InputSourceDataInputStream(dataStream);
SAXHandler contentHandler = new SAXHandler();
xmlReader.setContentHandler(contentHandler);
Boolean keepParsing = true;
while (keepParsing && is.hasData()) {
contentHandler.reset();
xmlReader.parse(is);
ParseResult pr = xmlReader.getProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_PARSERESULT());
...
keepParsing = !pr.isError();
}
Unparse
DataProcessor Unparse
The sameDataProcessor used for parse can be used to unparse an
infoset via the DataProcessor.unparse(org.apache.daffodil.api.infoset.InfosetInputter, java.nio.channels.WritableByteChannel) method. An InfosetInputter
provides the infoset to unparse, with the unparsed data written to the
provided WritableByteChannel. For example:
JDOMInfosetInputter jdomInputter = new JDOMInfosetInputter(doc);
UnparseResult ur = dp.unparse(jdomInputter, wbc)
SAX Unparse
In order to kick off an unparse via the SAX API, one must register theDaffodilUnparseContentHandler as the ContentHandler for an
XMLReader implementation. The call to the
DataProcessor.newContentHandlerInstance(java.nio.channels.WritableByteChannel)
method must be provided with the WritableByteChannel, where the unparsed
data ought to be written to. Any XMLReader implementation is permissible, as long as they have
XML Namespace support.
The call to the XMLReader.parse method must be wrapped in a try/catch, as the
DaffodilUnparseContentHandler relies on throwing an exception to
end processing in the case of any errors/failures. There are two kinds of exceptions it could throw:
DaffodilUnparseErrorSAXException- thrown when a processing error is encountered while unparsing. In this case,WithDiagnostics.isError()is true.DaffodilUnhandledSAXException- usually indicates a bug in Daffodil. This is an unchecked RuntimeException and should usually not be explicitly caught--it should be handled like one would handle any other RuntimeException. Note that if this is thrownDaffodilUnparseContentHandler.getUnparseResult()returns null.
DaffodilUnparseContentHandler.finish() method should be called--this
allows content handler state to be cleaned up and ensures the
DaffodilUnparseContentHandler.getUnparseResult() returns an
UnparseResult even in cases where the XMLReader encounters an
error. For example:
InputSource input = ...
OutputStream output = ...
WritableByteChannel wbc = Channels.newChannel(output);
DaffodilUnparseContentHandler unparseContentHandler = dp.newContentHandlerInstance(wbc);
XMLReader xmlReader = ...
xmlReader.setContentHandler(unparseContentHandler)
try {
xmlReader.parse(input);
} catch (DaffodilUnparseErrorSAXException e) {
// generally can be ignored, use getUnparseResult() instead
} catch (SAXException e) {
// non-Daffodil related exceptions created by the XMLReader, for example invalid XML
...
} finally {
unparseContentHandler.finish();
}
UnparseResult ur = unparseContentHandler.getUnparseResult();
Failures and Diagnostics
It is possible that failures could occur during the creation of theProcessorFactory, DataProcessor,
ParseResult, or UnparseResult. However,
rather than throwing an exception on error (e.g. invalid DFDL schema, parse error, etc), these classes extend
WithDiagnostics, which is used to determine if an error occurred,
and any diagnostic information (see Diagnostic) related to the step.
Thus, before continuing, one must check WithDiagnostics.isError().
For example:
ProcessorFactor pf = c.compile(files);
if (pf.isError()) {
java.util.List<Diagnostic> diags = pf.getDiagnostics();
for (Diagnostic d : diags) {
System.out.println(d.toString());
}
return -1;
}
Saving and Reloading Parsers
In some cases, it may be beneficial to save a parser and reload it. For example, when starting up, it may be quicker to reload an already compiled parser than to compile it from scratch. To save aDataProcessor:
DataProcessor dp = pf.onPath("/");
dp.save(saveFile);
And to restore a saved DataProcessor:
DataProcessor dp = Daffodil.reload(saveFile);
And use like below:
ParseResult pr = dp.parse(data);
or
DaffodilParseXMLReader xmlReader = dp.newXMLReaderInstance();
... // setting appropriate handlers
xmlReader.parse(data);
ParseResult pr = xmlReader.getProperty(DaffodilParseXMLReader.DAFFODIL_SAX_URN_PARSERESULT());
Thread Safety
A compiled DFDL schema typically has a far larger memory footprint than the data it parses, so it is usually worth compiling a schema once and sharing the result with as many threads as needed. In short, share the compiled schema, but do not share anything that belongs to a single parse or unparse.The objects of this API fall into two groups: the immutable objects that describe the compiled schema, which may be shared by any number of threads, and the stateful objects that belong to a single parse or unparse, which must not be.
Objects That May Be Shared Between Threads
The following objects are immutable once created. Any number of threads may hold and call them at the same time, with no additional synchronization:Daffodil- every static factory method of this class is thread safe and may be called from any thread.CompilerProcessorFactoryDataProcessorParseResultandUnparseResult
Compiler.withTunable(String, String),
ProcessorFactory.withDistinguishedRootNode(String, String),
DataProcessor.withValidation(String, java.net.URL), and every
other withX method returns a new object. One thread may therefore create a new
configuration while other threads keep using the original.
The methods that do the actual work,
DataProcessor.parse(org.apache.daffodil.api.InputSourceDataInputStream, org.apache.daffodil.api.infoset.InfosetOutputter)
and
DataProcessor.unparse(org.apache.daffodil.api.infoset.InfosetInputter, java.nio.channels.WritableByteChannel),
may be called concurrently on a shared DataProcessor, mixing
parses and unparses in any combination. Everything that belongs to a single run, such as the bit
position in the data, the DFDL variables, the infoset being built, and the diagnostics, is created
fresh for each call, so concurrent runs cannot see or change each other's state. The same is true
of DataProcessor.newXMLReaderInstance(),
DataProcessor.newContentHandlerInstance(java.nio.channels.WritableByteChannel),
DataProcessor.save(java.nio.channels.WritableByteChannel),
DataProcessor.walkMetadata(org.apache.daffodil.api.metadata.MetadataHandler),
ProcessorFactory.onPath(String), and the
compileX and reload methods of Compiler. Each call
to one of these creates independent state, so a single DataProcessor shared by a pool of
threads is the normal, recommended way to use Daffodil.
Objects That Must Be Unique To Each Thread
The following objects hold state for the duration of a run. They are not thread safe; each thread must use its own instance, and an instance must not be given to another thread until every thread that used it is finished with it:InputSourceDataInputStream- the current position in the data lives in this object, so each parse needs its own. One shared instance would also interleave reads of the underlyingInputStream.InfosetOutputterandInfosetInputter, and therefore the infosets they build or read. AnInfosetOutputtermay be reused by a thread, one parse at a time, after callingInfosetOutputter.reset().DaffodilParseXMLReaderand the handlers, resolvers and properties registered on it. A thread that wants to parse data with SAX needs its own reader, which is whyDataProcessor.newXMLReaderInstance()is thread safe.DaffodilUnparseContentHandler, for the same reason.Debugger- a debugger is interactive and keeps track of where the run it is attached to stopped, so aDataProcessorconfigured with a debugger should be confined to one thread.CodeGenerator
WritableByteChannel given to
DataProcessor.unparse(org.apache.daffodil.api.infoset.InfosetInputter, java.nio.channels.WritableByteChannel),
or the OutputStream given to one of the infoset text outputters, must be unique to
that call.
Objects that Daffodil calls back into have their own requirements, documented with them. The
Validator and
ValidatorFactory Service Provider Interface
implementations must be thread safe, because a single validator is used by all of the concurrent
parses that were configured with it. A DaffodilEntityResolverProvider
must also be thread safe, but the resolvers it returns are only ever used by one thread at a time.
Sharing Results Between Threads
AParseResult or UnparseResult is
only complete, and so only readable, once the call that returned it has finished. Passing the
result to another thread therefore needs the usual handoff, for example an
ExecutorService and a Future, which also
provides the memory visibility that the reading thread needs. The same applies to any infoset
built by an InfosetOutputter: build it on one thread, hand
it off, then read it on another.
Multi-threaded Example
// compile once, on the main thread, and share the compiled schema with all of the workers
Compiler c = Daffodil.compiler();
ProcessorFactory pf = c.compileFile(schemaFile);
if (pf.isError()) { ... } // no other thread is running yet, so nothing waits on the compilation
DataProcessor dp = pf.onPath("/");
ExecutorService pool = Executors.newFixedThreadPool(4);
for (final File dataFile : dataFiles) {
pool.submit(() -> {
// both of these belong to this parse alone and are never seen by another thread
try (InputSourceDataInputStream input =
Daffodil.newInputSourceDataInputStream(new FileInputStream(dataFile))) {
JDOMInfosetOutputter output = new JDOMInfosetOutputter();
ParseResult pr = dp.parse(input, output); // thread safe, dp is shared by all four threads
if (pr.isError()) {
...
} else {
Document doc = output.getResult();
...
}
} catch (IOException e) {
...
}
});
}
-
ClassDescriptionSource code generation and compilation is performed with a language-specific
CodeGenerator}, which must be interrogated for diagnostics to see if each call was successful or not.Compile DFDL schemas intoProcessorFactory's or reload saved parsers intoDataProcessor's.Factory object to create aCompilerThese are the events that a derived specific InfosetInputter creates.Implement this interface and register it with SPI, by listing the implementation in META-INF/services/org.apache.daffodil.api.DaffodilEntityResolverProvider, to replace the resolvers that Daffodil uses to resolve import/include schemaLocations.SAX Method of parsing schema and getting the DFDL Infoset via designated org.xml.sax.ContentHandler, based on the org.xml.sax.XMLReader interfaceAccepts SAX callback events from any SAX XMLReader for unparsingReturns the EntityResolver used by Daffodil to resolve import/include schemaLocations.Relevant data location for a diagnostic message.Compiled version of a DFDL Schema, used to parse data and get the DFDL infosetAn enumeration of all DFDL's simple types.Class containing diagnostic information.Indicates that this API is experimental and may change or be removed in the future.Provides Daffodil with byte data from an InputStream, ByteBuffer, or byte Array.Relevant schema location for a diagnostic message.Result of callingDataProcessor.parse(input:org\.apache\.daffodil* DataProcessor.parse, containing any diagnostic information, and the final data locationFactory to createDataProcessors, used for parsing dataInterface for Parse and Unparse resultsResult of callingDataProcessor#unparse(InfosetInputter, java.nio.channels.WritableByteChannel), containing diagnostic informationInterface that adds diagnostic information to classes that extend it.