Panic followed the inspectors into the flooded neighborhoods.

Within weeks of the August flood, the federal government estimated that the homes of thousands of people in South Louisiana had been "substantially damaged" -- the bureaucrats' phrase for describing a building rocked so badly the repairs would cost at least half its value. In practical terms, it is a designation that could force owners to elevate, wind up knocking their houses down and building on higher ground, or, in some cases, abandoning their homes altogether. 

But, so far, thousands of homes jacked up into the sky haven't appeared. Some early estimates were probably wrong. Elsewhere, building officials suspect some people are working outside the rules, and, at least in one parish, it's not clear if all homeowners know about their initial findings. 

As local authorities checked the federal contractors' work, many homes fell off the substantial damage lists. For example, Baton Rouge city-parish officials reported in December that FEMA had estimated 6,000 of the 48,000 flooded structures may have been substantially damaged. The city-parish doesn't make its own assessment until people come in an apply for a building permit, but the number has been slowly shrinking as properties are cleared. As of mid-January, the number of properties flagged by FEMA stood at 3,250, while only 194 properties have been confirmed as substantially damaged. 

After the flood in places like Ascension Parish, Central and Denham Springs, local officials made visits or sent out letters to families whose houses were thought to be substantially damaged. But in Baton Rouge, the city-parish put the onus on property owners, expecting that they would find out if their property was damaged at that level when they sought permits for repairs. Officials assumed they could inform residents when they applied for building permits, but people haven't always come asking for permission to rebuild. Livingston Parish, too, came up with a different system, as the local building official is performing all the assessments himself without federal contractors.

Substantial damage is one of the tent poles around which the entire flood insurance system is based, with the elevation requirements meant to protect both the entire program and individual homeowners. The program doesn't want to see properties flood repeatedly. Owners, whose insurance premiums can rise if they don't meet elevation requirements, face decisions between taking a financial hit in the short term or getting bled over the years, particularly when they try to sell their property.

Those thousands of Baton Rouge properties identified by FEMA but unverified by the city-parish remain a mystery. Development Director Carey Chauvin expects that many people are just doing off-the-books repairs, even to properties flagged by federal contractors.

"We know that's happening ... we just don't have the manpower to check," he said.

Chauvin said it is the responsibility of property owners to follow the permitting process already in place so they can be assessed for substantial damage. His department is "being very vigilant" in checking the appraisals, contacts and other material brought by people who come in for their permits and inspections, he said.

Property owners are supposed to apply for permits whenever they do work on water or gas lines, any time they replace an entire wall of drywall or machinery like a heating and air unit, plus electrical work of just about any kind. However, based on the number of outstanding cases, people in the city-parish — who didn't receive substantial damage letters — could in some cases being doing this kind of work without permits.

Not all houses that are damaged more than 50 percent of their value will need to be elevated or rebuilt. The requirement applies only for those properties in FEMA's high risk flood zones, also known as A zones and special flood hazard areas. Houses that already meet base flood elevation requirements also won't be forced to change, though inspectors in most areas attempted to target places where those exceptions wouldn't apply.  

Last month, Central officials reported that only 39 substantial damage estimates had been confirmed, while 347 had been reversed and another 411 were pending. As of Friday, 236 more cases had been cleared with only three more substantial damage confirmations. According to city officials, most changes came because contractors offered new repair estimates, rather than owners showing that they already reached elevation requirements.

As of Friday, FEMA had identified 2,329 structures as potentially substantially damaged in Denham Springs. Of those, 1,619 had been appealed and 1,529 of those appeals resulted in the substantial damage determination being dismissed, with another 49 challenges pending. It's unclear why the FEMA estimates were overturned, because the city doesn't track why a case was thrown out. 

It's difficult to compare the substantial damage numbers across communities because cities and parishes use different methods, as is their prerogative. Ascension Parish used land records to specifically target older neighborhoods where homes are less likely to be in compliance with current building standards and wound up identifying 1,556 homes that received substantial damage letters. Residents have filed 179 appeals and so far 142 substantial damage determinations have been overturned. Of those, 114 changed substantial to non-substantial damage, while 28 went from non-substantial to substantial damage.

Denham Springs also sent letters -- 3,114 in the first round. However, with all the flood damage, hundreds were returned, so authorities eventually resorted to knocking on doors and stapling notices on buildings thought to be substantially damaged. Central officials also hit the streets to follow-up on flagged properties where owners hadn't submitted contractors' estimates.

August's storm was so massive that it flooded buildings that are already built to what the government considers reasonable standards. Most communities require structures in high-risk floodplains to elevate to the level of the type of storm that has a one percent chance of occurring each year; in Baton Rouge, new buildings have to go up an additional foot. Houses in lower-risk areas, also known as X zones, don't have to follow substantial damage requirements.

Some houses were still left with substantial damage, frequently on homes built before the Federal Emergency Management Agency implemented its flood maps, which were introduced to Baton Rouge in July 1979, then meaningfully updated in 1993 and 2008 to become more stringent. Residents could submit a property appraisal and a contractor's estimate for the cost of repairs to try to lower their damage assessment below the 50 percent threshold. Those efforts are worthwhile, building officials said, because the federal contractors only performed cursory inspections of flooded homes, often estimating houses to be substantially damaged without going inside.

Yet it is possible for property owners to skirt the rules by submitting inflated appraisals or artificially low repair costs. Local authorities, who ultimately make substantial damage determinations, have said they've spot-checked a few homes, but by and large they don't have the resources to make sure the quotes are honest.

Denham Springs building official Rick Foster believes some cases are slipping through. He suspects it's a minority but remarked that it's not hard to fight an unwanted substantial damage claim.

"My job is not to audit every single contractor who comes in with a quote. ... I don't have the time for that," he said.

Officially, FEMA encourages people to elevate flood-vulnerable property. As the administrator of the National Flood Insurance Program, it has a vested interest in ensuring that the system works, and FEMA has the power to audit communities where flood insurance is available.

However, those audits are not public record, and building officials have said they're mostly concerned with faithfully recording the numbers property owners bring them, not following up to make sure they're correct or going out looking for people doing unauthorized construction out of sight. FEMA may sanction communities that don't enforce the rules -- including elevation. It could raise insurance fees or prevent an entire community from receiving flood insurance altogether, but it's never taken those steps in Louisiana.

Oil City and Rodessa, both in Caddo Parish, have been suspended from the NFIP, but they were removed because they refused to adopt new flood maps, not for a failure to enforce the flood maps already in place. Staff at the regional FEMA office, which includes Texas, New Mexico, Oklahoma, Arkansas and Louisiana, couldn't remember placing any communities on suspension or probation over lax enforcement for at least the last decade.

"The communities do a good job staying compliant," said Mark Lujan, senior regional insurance specialist.

The NFIP is playing a long game. Though it's known primarily for providing insurance payouts in the immediate aftermath of a flood, it is also designed to mold communities to higher building standards over time.

"The NFIP has always been more than insurance. It is focused on reducing the risk of flood in communities," said director Roy Wright.

Property owners in high-risk areas who receive federal assistance are required to enroll in the NFIP. Many people in those areas are already required by their mortgage holders to carry flood insurance, though people who own their houses outright are not obligated to keep their coverage.

There are penalties for insurance policies on new buildings that don't meet elevation requirements, just as there are some financial incentives for elevating higher than required. In the past, people who owned property built before FEMA flood maps were in place were eligible for subsidized coverage, but in the last five years, new laws have sought to raise rates to be more reflective of the true coverage cost.

Homeowners whose coverage has lapsed will pay much higher rates than they might remember before 2012. Since that year, premiums have been steadily rising on existing policies to reach the true cost. Rates increase five to 18 percent each year for most property, but 25 percent per year for property that's been repeatedly damaged. Under a 2014 law, if the owner sells or bequeaths  their property the new owner can keep the existing rate, though it will continue to rise. Properties vary, but a home may reach the true insurance cost in five to seven years if it's close to existing standards, whereas one that's severely out of code could see an annual increase for more than 15 years.

It's this long-term consequence — along with the threat of a repeat flood down the road — that has experts clamoring for people to elevate, even as they come out hundreds at a time to contest substantial damage estimates. After a flood, money is available to offset the cost of elevation, but it isn't enough to raise most homes, and it comes years down the road as a reimbursement rather than an immediate, surefire incentive.

Foster, the Denham Springs official, is putting a second floor on his own house and using the ground level for things like parking. His home was damaged somewhere in the 60 to 70 percent range, and he figures he could have successfully appealed, but he and his wife don't want to suffer in the floods to come, even if it means taking out another mortgage to pay for the work. He's advising his at-risk neighbors to follow his lead, even though his message isn't popular.

"I damn sure don't want people to flood again. It's my job," he said. "Elevate it, relocate it, do whatever you need to do. I don't want to see you flood again."

Staff writer David Mitchell contributed to this report.

Follow Steve Hardy on Twitter, @SteveRHardy.