GeoDirectory already works this way. A user can edit a listing from the front end only if they are its owner (the listing’s author), and admins can edit everything. What you need to do is make each owner the author of their listing.

Assign listings to their owners

1. Create a user account for each owner, or ask them to register. Any role that can log in works.
2. In wp-admin, edit the listing.
3. In the Place Owner box, search for the user and select them.
4. Update the listing.

Let owners claim their listing

With many listings, the Claim Listings add-on saves you the manual work. Owners click a claim link on their listing, you approve the claim, and the listing is assigned to their account.

Where owners edit

Once logged in, owners see an edit link on their own listing page from the GD > Author Actions block. The GD > Dashboard block, placed on any page, gives each user links to their own listings. Neither shows edit options for listings owned by someone else.

For the editing and deleting options in more detail, see how can my users edit or delete their listings.