Semi-Structured Data
Structure and the database boundary
A textbook has clear structure: chapters, sections, figures, cross-references, a table of contents. It would be prohibitively expensive to manually index every word and every structural relationship for database storage. The question is: what can sensibly, or usefully, be stored in a database?
Two broad approaches exist:
| Approach | Description | Query language |
|---|---|---|
| Store in two parts | Keep the document in its native form (LaTeX, Word, PDF); store indexable features in relational tables | SQL |
| Store once, perhaps shredded | Keep the document largely in native form (XML, JSON); develop database tools that navigate semi-structured data | XPath, XQuery, application-specific |
The second approach requires the database to return best-effort query answers, since schema violations could be frequent. A relational system would simply reject non-conforming data; a semi-structured system must cope.
Adding structure to unstructured documents
When faced with an unstructured corpus, structure can be imposed by:
- Human curators — annotate, classify, correct spellings, remove noise.
- Automatic tools — keyword extraction, named-entity recognition, classification, indexing.
The document is carved up and marked up for storage. The original can be reconstructed (un-shredded) as and when necessary.
Simple approach: keyword-based analysis — bag-of-words indexing, TF-IDF ranking, inverted indices. Fast, well-understood, limited semantic understanding.
Advanced approach: LLM or NLP-based analysis — entity linking, sentiment analysis, topic modelling, summarisation. More powerful but computationally expensive.
NLP techniques are not examinable for this course.
Semi-structured vs. unstructured
| Structured | Semi-structured | Unstructured | |
|---|---|---|---|
| Schema | Rigid, enforced | Flexible, implicit, or absent | None |
| Query | Precise (SQL) | Best-effort | Full-text search |
| Examples | RDBMS tables | XML, JSON documents | Raw text, images, audio |
Why this matters for databases
Semi-structured data sits between the rigid relational world and the chaos of unstructured text. It acknowledges that real-world data is messy: optional fields, repeated elements, heterogeneous records, evolving schemas. A semi-structured database accepts this messiness and provides tools to navigate it, rather than demanding that data be cleaned and normalised before ingestion.
Summary
- Not all data fits neatly into relational tables; some documents are too complex or too loosely structured.
- Two strategies: (1) pair native documents with relational metadata, (2) store in a flexible format (XML/JSON) with specialised query tools.
- Structure can be added by humans or automated tools.
- NLP techniques are out of scope for this course.