## Overview The Real Estate Alerter is a collaborative project developed as part of a journalism fellowship, bringing together journalists and technologists from multiple news organizations: Hearst and Gannett in the United States, The Globe and Mail in Canada, and e24 in Norway. The project aims to solve a fundamental challenge in real estate journalism: with thousands of property transactions occurring daily, how can newsrooms systematically identify the ones that are actually worth covering? The origin story illustrates the problem well. A Detroit Free Press editor, Randy Essex, happened to spot a "For Sale" sign for a $2 million property near Michigan train station while out running. He realized this was a significant story that no one had picked up. The resulting article became one of the most-read stories that week, prompting team member Annette to question whether there might be a more efficient approach than physically running around the city to find newsworthy transactions. ## System Architecture and Technical Approach The Real Estate Alerter employs a multi-stage pipeline that combines traditional data science techniques with LLM-based reasoning. The architecture consists of several key components working together: **Anomaly Detection Layer**: The first stage uses domain knowledge, statistical analysis, and clustering algorithms to identify outliers within the transaction dataset. This preprocessing step is crucial because it dramatically reduces the volume of data that needs to be processed by the more computationally expensive LLM stage. Rather than asking the LLM to evaluate every transaction, the system first filters down to transactions that are statistically unusual in some way. **Data Preprocessing and Enrichment**: Structured transaction data from databases is transformed into natural language format to be more easily digestible by the LLM. The team also enriches this data with external contextual information, such as details about geographical areas mentioned in the database, what those areas are known for, and what would be considered unusual about them. This contextual enrichment helps the LLM make more informed decisions about newsworthiness. **LLM-Based Newsworthiness Assessment**: The core of the system relies on an LLM to determine which of the identified outliers are actually newsworthy. The team employs few-shot prompting to direct the LLM's behavior, providing examples that illustrate what constitutes a newsworthy transaction. This is a critical design decision that acknowledges the LLM needs guidance—it cannot independently determine what journalists consider newsworthy. **Famous Person Detection**: A notable feature is the integration of named entity recognition on archive data to extract and maintain a sorted list of famous people. This list enriches the feature set used for newsworthiness detection. The team uses Wikidata to resolve challenges like matching stage names to legal names (e.g., querying "Lady Gaga" returns her legal name) and obtaining dates of birth to disambiguate common names (e.g., distinguishing a famous "James Smith" from many others with the same name). In Norway, they also leverage tax registry data for date of birth matching. ## Human-in-the-Loop Design A significant aspect of the LLMOps approach is the embedded human feedback mechanism. The team explicitly built the system to incorporate human judgment for several purposes: **False Positive Handling**: When the LLM flags a transaction as newsworthy but journalists determine it is not (such as the example of a property transaction flagged because "Christian Erikson" appeared—a famous Danish footballer's name that also happens to be a common Norwegian name), users can provide feedback through the dashboard. This feedback is collected for later fine-tuning of the system. **False Negative Review**: Recognizing that LLMs might miss genuinely newsworthy transactions, the team built a filter for reviewing transactions deemed "non-newsworthy" by the system. Journalists can quickly scan these to catch any stories that were incorrectly filtered out. **Cold Start Problem**: The team candidly acknowledges the chicken-and-egg problem: they needed to start the system before having human feedback data to train on. Their solution was to conduct interviews with real estate reporters, presenting them with feature lists and asking them to weight features based on how much they contribute to newsworthiness. This qualitative research informed their initial few-shot prompting approach. ## Defining Newsworthiness: The Core Challenge The team emphasizes that answering "what makes a transaction newsworthy?" was both the central challenge and the key to success. They discovered that this question doesn't have a universal answer: - What's newsworthy varies by region (contaminated ground is unusual in some markets but common in Oslo) - Standards differ between local and national newspapers - Even within the project team, reaching consensus was difficult The team made a pragmatic decision to focus on two features that emerged clearly from their reporter interviews: transaction price and involvement of famous persons. These became the primary signals for their few-shot prompting, with a rule-based backup algorithm in place to catch transactions meeting these criteria in case the LLM misses them. Interestingly, they observed that as more human feedback accumulated, the LLM began missing fewer of these clearly newsworthy transactions, suggesting the human-in-the-loop approach was working. ## Technical Challenges and Lessons Learned **Noisy Data**: When the team gained access to a richer dataset, they initially expected better results. Instead, they faced significant data cleaning challenges, and the results weren't proportionally better. Their hypothesis is that more complex feature sets require more sophisticated prompting or fine-tuning to avoid confusing the LLM. **Name Matching Complexity**: The famous person detection feature faces multiple challenges: - Celebrities often use stage names rather than legal names that appear in property records - Common names require additional disambiguating information like date of birth - The system once surfaced a "Julia Roberts" transaction in Oslo that turned out not to be the famous actress **Context-Dependent Rules**: The example of ground contamination is instructive for LLMOps practitioners. The LLM initially flagged contaminated properties as newsworthy—a reasonable assumption in many contexts—but this was actually common in the Oslo dataset and not particularly newsworthy. This illustrates why LLMs need domain-specific direction rather than relying on general world knowledge. ## Delivery Mechanism The system delivers alerts via Slackbot integration, notifying journalists when new newsworthy transactions are detected. Users can then access a dashboard to review transactions, see the LLM's description of why each transaction was flagged as newsworthy, and filter by criteria like celebrity involvement, price per square meter, or total transaction price. ## Current Status and Future Directions The team is transparent that the system is still in prototype stage—described candidly as "tape and glue"—and not yet fully operational in newsrooms. Several future improvements are planned: - Setting up a proper production pipeline and alert system - Scaling from Norwegian data to US and Canadian datasets (with noted differences in data availability and privacy considerations) - Making the system adaptable to different newsworthiness standards (local vs. national publications) - Building a UI for journalists to access and edit the famous person list - Expanding from individual transaction analysis to pattern detection (e.g., tracking when people or companies buy multiple properties over time) - Exploring whether the approach can be applied to other datasets beyond real estate ## Data Availability Considerations The team notes significant variation in data accessibility by geography. Norway has open data that the team purchases as daily transaction lists from a company called Ambita. In the US, data is available but typically at the state level and more expensive. In Canada, data access is particularly difficult. The team hopes that demonstrating journalistic impact with this tool might help advocate for greater data openness. ## Honest Assessment This case study represents an honest, early-stage project rather than a polished production deployment. The team's transparency about challenges, the "tape and glue" state of the system, and the need for continued improvement is refreshing. The approach of combining statistical anomaly detection with LLM-based reasoning and human feedback loops represents a sensible architecture for this type of application. However, the project has not yet demonstrated production-scale results, and many of the claimed benefits remain aspirational rather than proven. The example of catching a cross-country skiing star's property sale "with test data" (meaning they would have caught it, but didn't actually have the system running at the time) illustrates the gap between prototype promise and operational reality.