Combine all ingredients, except mangoes, in a bowl, lightly toss, cover with plastic and let it rest in the refrigerator for 30 minutes. Keep diced mangoes chilled in the refrigerator in a separate, covered container. Only add mangoes to this mix, gently toss and serve with the grilled swordfish right before serving.
Preheat grill to medium-high. Rub surface of fillets with olive oil and season with salt and pepper. Let it rest in the refrigerator for at least 5 minutes. Then, arrange fillets on grill rack, cover, and grill until flesh is opaque, 4 to 5 minutes per side. Slide fillets off grill and onto a platter or individual plates. Top with some mango salsa, letting the some of the mango salsa cascade down the swordfish fillet onto the plate. May be served with steamed rice or a steamed vegetable medley.