As far as i know, In the WoW-API the target nearest functions are protected (meaning they cannot be called from Addon codes). What you can do however, is let ur addon create a macro and let that run whenever you want.
To target the nearest enemy (npc) in a macro:
/targetenemy
(this is the same as pushing TAB on ur keyboard)
if you want to target an friendly npc, u can use
/targetfriend
(same as CTRL+TAB)
if you however want to target players instead of npcs, u'd have to use.
/targetenemyplayer
/targetfriendplayer
They work in the same fashion, just for players instead of npcs.
Do note; each of these commands may only be called ONCE per macro. If you want to check for both NPCs and Players, you'd have to call 2 seperate macros to do so.