A customer is facing a security challenge in their Next.js application and is seeking help to strengthen their defenses. Here's a summary of their situation:

1. Current Setup:

2. Using @edge-csrf/nextjs for CSRF protection

3. CSRF token is generated in middleware

4. Token is passed to the client via headers and cookies

5. The Problem:

6. Python scrapers are still able to get 200 responses

7. Copied requests from the network tab can be successfully tested

8. What They've Tried:

9. Implementing dynamic CSRF token generation

10. Adding custom headers to detect scraping attempts

11. What They Need:

12. A robust solution to block scraping attempts

13. Ensure CSRF protection is effective against automated tools

14. Next.js-specific best practices for this security scenario

The user has provided code snippets for their middleware, search page, and client-side form submission. They are looking for ideas to enhance their security measures and prevent unauthorized access to their application's data, particularly from automated scraping tools.

See below for code snippets:

middleware.ts

const csrfProtect = createCsrfProtect({  
  cookie: {  
    secure: process.env.NODE_ENV === "production",  
    sameSite: "strict",  
  },  
});  
  
// Next.js middleware function  
export const middleware = async (request: NextRequest) => {  
  const response = NextResponse.next();  
  
  try {  
    await csrfProtect(request, response);  
  } catch (err) {  
    if (err instanceof CsrfError)  
      return new NextResponse("invalid csrf token", { status: 403 });  
    throw err;  
  }  
  
  return response;  
};  

search/page.tsx

export default function SearcPage() {  
  const csrfToken = headers().get("X-CSRF-Token") || "missing";  
  return <SearchClientPage csrfToken={csrfToken} />;  
}  

SearchClientPage .tsx

const handleSubmit = async (event: React.FormEvent<HTMLFormElement>) => {  
    event.preventDefault();  
    let formData = new FormData(event.currentTarget);  
    formData.set("csrf_token", csrfToken);  
    formData.set("from", "search");  
    formAction(formData);  
  };  

Response from copy request from network tab and test it with PowerShell or py

1:{"success":true,"message":"­","data":{.....data}}}