HPE Storage Users Group

Hello guys,

We have an old 3par (no support) and we have a degraded disk which I haven't been able to remove. Servicemag operation fails everytime with the following:
3PAR cli% servicemag status -d
Cage 0, magazine 11:
A servicemag start command failed on this magazine.
The command completed at Fri Jun 10 15:31:01 2022.
The output of the servicemag start was:
servicemag start 0 11
... servicing disks in mag: 0 11
... normal disks:
... not normal disks: WWN [5000C50071808294] Id [11] diskpos [0]
.................... WWN [5000CCA0578D71CB] Id [12]
... relocating chunklets to spare space...
Failed --
Could not find relocation target for one or more chunklets
servicemag start 0 11 -- Failed

I've found some procedures over the forums and communities and eventually some chunklets were removed but I still get the "Could not find relocation target for one or more chunklets":
3PAR cli% movepdtospare -f -vacate 11
Error: Could not find relocation target for one or more chunklets
3PAR cli% movech -perm -ovrd 11:0
Are you sure you want to move the chunklets?
select q=quit y=yes n=no: y
Error: Disk 11 chunklet 0 is not in use.
3PAR cli% showpdch -spr 11
No chunklet information available.
3PAR cli% removespare 11:a
Are you sure you want to remove spares?
select q=quit y=yes n=no: y
No spares removed
3PAR cli% dismisspd 11
Error : Pd id 11 is in use

I tried just removing the disk and then dismisspd 11 but result is
3PAR cli% dismisspd 11
Error : Pd id 11 is in use

any ideas?

what does:



say ?

MammaGutt wrote:what does:

Code: Select all

showpd -c 11
showpdvv 11
showpdld 11
showpdch 11

say ?

This is the result:
3PAR cli% showpd -c 11
------- Normal Chunklets -------- ---- Spare Chunklets ----
- Used - -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State Total OK Fail Free Uninit Unavail Fail OK Fail Free Uninit Fail
11 0:11:0 FC degraded 819 0 283 0 468 0 68 0 0 0 0 0
------------------------------------------------------------------------------------------
1 total 819 0 283 0 468 0 68 0 0 0 0 0
3PAR cli% showpdvv 11
PDId CagePos Type RPM VVId VVName VVSp
11 0:11:0 FC 10 26 DataStore3PAR01.0 usr
11 0:11:0 FC 10 27 DataStore3PAR01.1 usr
11 0:11:0 FC 10 33 DataStore3PAR02 usr
11 0:11:0 FC 10 35 DataStore3PAR01.2 usr
3PAR cli% showpdld 11
invalid command name "showpdld"
3PAR cli% showpdch 11
Pdid Chnk LdName LdCh State Usage Media Sp Cl From To
11 452 tp-1-sd-0.16 286 stale ld valid N N --- ---
11 453 tp-1-sd-0.18 4 stale ld valid N N --- ---
11 455 tp-1-sd-0.18 16 stale ld valid N N --- ---
11 456 tp-1-sd-0.18 22 logging ld valid N N --- ---
11 457 tp-1-sd-0.18 28 stale ld valid N N --- ---
11 459 tp-1-sd-0.18 40 stale ld valid N N --- ---
11 460 tp-1-sd-0.18 46 stale ld valid N N --- ---
11 461 tp-1-sd-0.18 52 stale ld valid N N --- ---
11 462 tp-1-sd-0.18 58 logging ld valid N N --- ---
11 463 tp-1-sd-0.18 64 logging ld valid N N --- ---
11 464 tp-1-sd-0.18 70 stale ld valid N N --- ---
11 465 tp-1-sd-0.18 76 normal ld valid N N --- ---
11 466 tp-1-sd-0.18 82 stale ld valid N N --- ---
11 467 tp-1-sd-0.18 88 stale ld valid N N --- ---
11 468 tp-1-sd-0.18 94 normal ld valid N N --- ---
11 469 tp-1-sd-0.18 100 normal ld valid N N --- ---
11 470 tp-1-sd-0.18 106 normal ld valid N N --- ---
11 471 tp-1-sd-0.18 112 stale ld valid N N --- ---
11 472 tp-1-sd-0.18 118 stale ld valid N N --- ---
11 473 tp-1-sd-0.18 124 stale ld valid N N --- ---
11 474 tp-1-sd-0.18 130 stale ld valid N N --- ---
11 475 tp-1-sd-0.18 136 stale ld valid N N --- ---
11 476 tp-1-sd-0.18 142 stale ld valid N N --- ---
11 477 tp-1-sd-0.18 148 stale ld valid N N --- ---
11 478 tp-1-sd-0.18 154 stale ld valid N N --- ---
11 479 tp-1-sd-0.18 160 stale ld valid N N --- ---
11 480 tp-1-sd-0.18 166 stale ld valid N N --- ---
11 481 tp-1-sd-0.18 172 normal ld valid N N --- ---
11 482 tp-1-sd-0.18 178 stale ld valid N N --- ---
11 483 tp-1-sd-0.18 184 stale ld valid N N --- ---
11 484 tp-1-sd-0.18 190 stale ld valid N N --- ---
11 485 tp-1-sd-0.18 196 stale ld valid N N --- ---
11 486 tp-1-sd-0.18 202 stale ld valid N N --- ---
11 487 tp-1-sd-0.18 208 stale ld valid N N --- ---
11 488 tp-1-sd-0.18 214 stale ld valid N N --- ---
11 489 tp-1-sd-0.18 220 stale ld valid N N --- ---
11 490 tp-1-sd-0.18 226 stale ld valid N N --- ---
11 491 tp-1-sd-0.18 232 stale ld valid N N --- ---
11 492 tp-1-sd-0.18 238 stale ld valid N N --- ---
11 493 tp-1-sd-0.18 244 stale ld valid N N --- ---
11 494 tp-1-sd-0.18 250 stale ld valid N N --- ---
11 495 tp-1-sd-0.18 256 stale ld valid N N --- ---
11 496 tp-1-sd-0.18 262 stale ld valid N N --- ---
11 497 tp-1-sd-0.18 268 stale ld valid N N --- ---
11 498 tp-1-sd-0.18 274 stale ld valid N N --- ---
11 499 tp-1-sd-0.18 280 stale ld valid N N --- ---
11 500 tp-1-sd-0.18 286 stale ld valid N N --- ---
11 501 tp-1-sd-0.20 4 stale ld valid N N --- ---
11 502 tp-1-sd-0.20 10 stale ld valid N N --- ---
11 503 tp-1-sd-0.20 16 stale ld valid N N --- ---
11 504 tp-1-sd-0.20 22 stale ld valid N N --- ---
11 505 tp-1-sd-0.20 28 stale ld valid N N --- ---
11 506 tp-1-sd-0.20 34 stale ld valid N N --- ---
11 507 tp-1-sd-0.20 40 normal ld valid N N --- ---
11 508 tp-1-sd-0.20 46 normal ld valid N N --- ---
11 509 tp-1-sd-0.20 52 stale ld valid N N --- ---
11 510 tp-1-sd-0.20 58 stale ld valid N N --- ---
11 511 tp-1-sd-0.20 64 stale ld valid N N --- ---
11 512 tp-1-sd-0.20 70 stale ld valid N N --- ---
11 513 tp-1-sd-0.20 76 stale ld valid N N --- ---
11 514 tp-1-sd-0.20 82 stale ld valid N N --- ---
11 515 tp-1-sd-0.20 88 stale ld valid N N --- ---
11 516 tp-1-sd-0.20 94 stale ld valid N N --- ---
11 517 tp-1-sd-0.20 100 stale ld valid N N --- ---
11 518 tp-1-sd-0.20 106 stale ld valid N N --- ---
11 519 tp-1-sd-0.20 112 stale ld valid N N --- ---
11 520 tp-1-sd-0.20 118 stale ld valid N N --- ---
11 521 tp-1-sd-0.20 124 stale ld valid N N --- ---
11 522 tp-1-sd-0.20 130 stale ld valid N N --- ---
11 523 tp-1-sd-0.20 136 stale ld valid N N --- ---
11 524 tp-1-sd-0.20 142 stale ld valid N N --- ---
11 525 tp-1-sd-0.20 148 normal ld valid N N --- ---
11 526 tp-1-sd-0.20 154 stale ld valid N N --- ---
11 527 tp-1-sd-0.20 160 normal ld valid N N --- ---
11 528 tp-1-sd-0.20 166 normal ld valid N N --- ---
11 529 tp-1-sd-0.20 172 normal ld valid N N --- ---
11 530 tp-1-sd-0.20 178 normal ld valid N N --- ---
11 531 tp-1-sd-0.20 184 stale ld valid N N --- ---
11 532 tp-1-sd-0.20 190 normal ld valid N N --- ---
11 533 tp-1-sd-0.20 196 normal ld valid N N --- ---
11 534 tp-1-sd-0.20 202 normal ld valid N N --- ---
11 535 tp-1-sd-0.20 208 normal ld valid N N --- ---
11 536 tp-1-sd-0.20 214 stale ld valid N N --- ---
11 538 tp-1-sd-0.28 238 stale ld valid N N --- ---
11 539 tp-1-sd-0.20 220 stale ld valid N N --- ---
11 540 tp-1-sd-0.20 226 stale ld valid N N --- ---
11 541 tp-1-sd-0.20 232 stale ld valid N N --- ---
11 542 tp-1-sd-0.20 238 stale ld valid N N --- ---
11 543 tp-1-sd-0.20 244 normal ld valid N N --- ---
11 544 tp-1-sd-0.20 250 normal ld valid N N --- ---
11 545 tp-1-sd-0.20 256 stale ld valid N N --- ---
11 546 tp-1-sd-0.20 262 stale ld valid N N --- ---
11 547 tp-1-sd-0.20 268 logging ld valid N N --- ---
11 548 tp-1-sd-0.20 274 stale ld valid N N --- ---
11 549 tp-1-sd-0.20 280 stale ld valid N N --- ---
11 550 tp-1-sd-0.20 286 stale ld valid N N --- ---
11 551 tp-1-sd-0.22 4 stale ld valid N N --- ---
11 552 tp-1-sd-0.22 10 stale ld valid N N --- ---
11 553 tp-1-sd-0.22 16 stale ld valid N N --- ---
11 554 tp-1-sd-0.22 22 stale ld valid N N --- ---
11 555 tp-1-sd-0.22 28 stale ld valid N N --- ---
11 556 tp-1-sd-0.22 34 stale ld valid N N --- ---
11 557 tp-1-sd-0.22 40 stale ld valid N N --- ---
11 558 tp-1-sd-0.22 46 stale ld valid N N --- ---
11 559 tp-1-sd-0.22 52 stale ld valid N N --- ---
11 560 tp-1-sd-0.22 58 stale ld valid N N --- ---
11 561 tp-1-sd-0.22 64 stale ld valid N N --- ---
11 562 tp-1-sd-0.22 70 normal ld valid N N --- ---
11 563 tp-1-sd-0.22 76 normal ld valid N N --- ---
11 564 tp-1-sd-0.22 82 normal ld valid N N --- ---
11 565 tp-1-sd-0.22 88 stale ld valid N N --- ---
11 566 tp-1-sd-0.22 94 stale ld valid N N --- ---
11 567 tp-1-sd-0.22 100 normal ld valid N N --- ---
11 568 tp-1-sd-0.22 106 normal ld valid N N --- ---
11 569 tp-1-sd-0.22 112 normal ld valid N N --- ---
11 570 tp-1-sd-0.22 118 normal ld valid N N --- ---
11 571 tp-1-sd-0.22 124 normal ld valid N N --- ---
11 572 tp-1-sd-0.22 130 stale ld valid N N --- ---
11 573 tp-1-sd-0.22 136 stale ld valid N N --- ---
11 574 tp-1-sd-0.22 142 stale ld valid N N --- ---
11 575 tp-1-sd-0.22 148 stale ld valid N N --- ---
11 576 tp-1-sd-0.22 154 stale ld valid N N --- ---
11 577 tp-1-sd-0.22 160 stale ld valid N N --- ---
11 578 tp-1-sd-0.22 166 logging ld valid N N --- ---
11 579 tp-1-sd-0.22 172 stale ld valid N N --- ---
11 580 tp-1-sd-0.22 178 stale ld valid N N --- ---
11 581 tp-1-sd-0.22 184 stale ld valid N N --- ---
11 582 tp-1-sd-0.22 190 stale ld valid N N --- ---
11 583 tp-1-sd-0.22 196 stale ld valid N N --- ---
11 584 tp-1-sd-0.22 202 stale ld valid N N --- ---
11 585 tp-1-sd-0.22 208 stale ld valid N N --- ---
11 586 tp-1-sd-0.22 214 stale ld valid N N --- ---
11 587 tp-1-sd-0.22 220 stale ld valid N N --- ---
11 588 tp-1-sd-0.22 226 normal ld valid N N --- ---
11 589 tp-1-sd-0.22 232 logging ld valid N N --- ---
11 590 tp-1-sd-0.22 238 stale ld valid N N --- ---
11 591 tp-1-sd-0.22 244 stale ld valid N N --- ---
11 592 tp-1-sd-0.22 250 logging ld valid N N --- ---
11 593 tp-1-sd-0.22 256 stale ld valid N N --- ---
11 594 tp-1-sd-0.22 262 stale ld valid N N --- ---
11 595 tp-1-sd-0.22 268 stale ld valid N N --- ---
11 596 tp-1-sd-0.22 274 stale ld valid N N --- ---
11 597 tp-1-sd-0.22 280 stale ld valid N N --- ---
11 598 tp-1-sd-0.22 286 stale ld valid N N --- ---
11 599 tp-1-sd-0.16 292 normal ld valid N N --- ---
11 600 tp-1-sd-0.16 298 normal ld valid N N --- ---
11 601 tp-1-sd-0.16 304 normal ld valid N N --- ---
11 602 tp-1-sd-0.28 256 stale ld valid N N --- ---
11 603 tp-1-sd-0.28 262 stale ld valid N N --- ---
11 604 tp-1-sd-0.28 268 stale ld valid N N --- ---
11 605 tp-1-sd-0.28 274 stale ld valid N N --- ---
11 606 tp-1-sd-0.28 280 stale ld valid N N --- ---
11 607 tp-1-sd-0.26 196 stale ld valid N N --- ---
11 608 tp-1-sd-0.26 202 stale ld valid N N --- ---
11 609 tp-1-sd-0.26 208 stale ld valid N N --- ---
11 610 tp-1-sd-0.26 214 stale ld valid N N --- ---
11 611 tp-1-sd-0.26 220 stale ld valid N N --- ---
11 612 tp-1-sd-0.26 226 stale ld valid N N --- ---
11 613 tp-1-sd-0.26 232 stale ld valid N N --- ---
11 614 tp-1-sd-0.26 238 stale ld valid N N --- ---
11 615 tp-1-sd-0.26 244 stale ld valid N N --- ---
11 616 tp-1-sd-0.26 250 stale ld valid N N --- ---
11 617 tp-1-sd-0.26 256 stale ld valid N N --- ---
11 618 tp-1-sd-0.26 262 stale ld valid N N --- ---
11 619 tp-1-sd-0.26 268 stale ld valid N N --- ---
11 620 tp-1-sd-0.26 274 stale ld valid N N --- ---
11 621 tp-1-sd-0.26 280 logging ld valid N N --- ---
11 622 tp-1-sd-0.26 286 stale ld valid N N --- ---
11 623 tp-1-sd-0.28 286 stale ld valid N N --- ---
11 624 tp-1-sd-0.28 292 stale ld valid N N --- ---
11 625 tp-1-sd-0.28 298 stale ld valid N N --- ---
11 626 tp-1-sd-0.24 4 stale ld valid N N --- ---
11 627 tp-1-sd-0.24 10 stale ld valid N N --- ---
11 628 tp-1-sd-0.24 16 stale ld valid N N --- ---
11 629 tp-1-sd-0.24 22 stale ld valid N N --- ---
11 630 tp-1-sd-0.24 28 stale ld valid N N --- ---
11 631 tp-1-sd-0.24 34 stale ld valid N N --- ---
11 632 tp-1-sd-0.24 40 normal ld valid N N --- ---
11 633 tp-1-sd-0.24 46 normal ld valid N N --- ---
11 634 tp-1-sd-0.24 52 stale ld valid N N --- ---
11 635 tp-1-sd-0.24 58 stale ld valid N N --- ---
11 636 tp-1-sd-0.24 64 stale ld valid N N --- ---
11 637 tp-1-sd-0.24 70 logging ld valid N N --- ---
11 638 tp-1-sd-0.24 76 stale ld valid N N --- ---
11 639 tp-1-sd-0.24 82 stale ld valid N N --- ---
11 640 tp-1-sd-0.24 88 stale ld valid N N --- ---
11 641 tp-1-sd-0.24 94 stale ld valid N N --- ---
11 642 tp-1-sd-0.24 100 stale ld valid N N --- ---
11 643 tp-1-sd-0.24 106 stale ld valid N N --- ---
11 644 tp-1-sd-0.24 112 stale ld valid N N --- ---
11 645 tp-1-sd-0.24 118 stale ld valid N N --- ---
11 646 tp-1-sd-0.24 124 normal ld valid N N --- ---
11 647 tp-1-sd-0.26 4 stale ld valid N N --- ---
11 648 tp-1-sd-0.26 10 logging ld valid N N --- ---
11 649 tp-1-sd-0.26 16 stale ld valid N N --- ---
11 650 tp-1-sd-0.26 22 stale ld valid N N --- ---
11 651 tp-1-sd-0.26 28 stale ld valid N N --- ---
11 652 tp-1-sd-0.26 34 stale ld valid N N --- ---
11 653 tp-1-sd-0.26 40 stale ld valid N N --- ---
11 654 tp-1-sd-0.26 46 stale ld valid N N --- ---
11 655 tp-1-sd-0.26 52 stale ld valid N N --- ---
11 656 tp-1-sd-0.26 58 stale ld valid N N --- ---
11 657 tp-1-sd-0.26 64 stale ld valid N N --- ---
11 658 tp-1-sd-0.26 70 stale ld valid N N --- ---
11 659 tp-1-sd-0.26 76 stale ld valid N N --- ---
11 660 tp-1-sd-0.26 82 stale ld valid N N --- ---
11 661 tp-1-sd-0.26 88 stale ld valid N N --- ---
11 662 tp-1-sd-0.26 94 normal ld valid N N --- ---
11 663 tp-1-sd-0.26 100 normal ld valid N N --- ---
11 664 tp-1-sd-0.26 106 normal ld valid N N --- ---
11 665 tp-1-sd-0.26 112 normal ld valid N N --- ---
11 666 tp-1-sd-0.26 118 stale ld valid N N --- ---
11 667 tp-1-sd-0.26 124 stale ld valid N N --- ---
11 668 tp-1-sd-0.26 130 stale ld valid N N --- ---
11 669 tp-1-sd-0.26 136 stale ld valid N N --- ---
11 670 tp-1-sd-0.26 142 stale ld valid N N --- ---
11 671 tp-1-sd-0.26 148 stale ld valid N N --- ---
11 672 tp-1-sd-0.26 154 normal ld valid N N --- ---
11 673 tp-1-sd-0.26 160 stale ld valid N N --- ---
11 674 tp-1-sd-0.26 166 stale ld valid N N --- ---
11 675 tp-1-sd-0.26 172 stale ld valid N N --- ---
11 676 tp-1-sd-0.26 178 stale ld valid N N --- ---
11 677 tp-1-sd-0.26 184 normal ld valid N N --- ---
11 678 tp-1-sd-0.26 190 logging ld valid N N --- ---
11 679 tp-1-sd-0.24 130 logging ld valid N N --- ---
11 680 tp-1-sd-0.24 136 stale ld valid N N --- ---
11 681 tp-1-sd-0.24 142 logging ld valid N N --- ---
11 682 tp-1-sd-0.24 148 normal ld valid N N --- ---
11 683 tp-1-sd-0.24 154 normal ld valid N N --- ---
11 684 tp-1-sd-0.24 160 normal ld valid N N --- ---
11 685 tp-1-sd-0.24 166 normal ld valid N N --- ---
11 686 tp-1-sd-0.24 172 normal ld valid N N --- ---
11 687 tp-1-sd-0.24 178 stale ld valid N N --- ---
11 688 tp-1-sd-0.24 184 stale ld valid N N --- ---
11 689 tp-1-sd-0.24 190 normal ld valid N N --- ---
11 690 tp-1-sd-0.24 196 normal ld valid N N --- ---
11 691 tp-1-sd-0.24 202 normal ld valid N N --- ---
11 692 tp-1-sd-0.24 208 stale ld valid N N --- ---
11 693 tp-1-sd-0.24 214 stale ld valid N N --- ---
11 694 tp-1-sd-0.0 298 stale ld valid N N --- ---
11 695 tp-1-sd-0.28 4 stale ld valid N N --- ---
11 696 tp-1-sd-0.28 10 stale ld valid N N --- ---
11 697 tp-1-sd-0.28 16 stale ld valid N N --- ---
11 698 tp-1-sd-0.28 22 stale ld valid N N --- ---
11 699 tp-1-sd-0.28 28 stale ld valid N N --- ---
11 700 tp-1-sd-0.28 34 logging ld valid N N --- ---
11 701 tp-1-sd-0.28 40 stale ld valid N N --- ---
11 702 tp-1-sd-0.28 46 logging ld valid N N --- ---
11 703 tp-1-sd-0.28 52 stale ld valid N N --- ---
11 704 tp-1-sd-0.28 58 stale ld valid N N --- ---
11 705 tp-1-sd-0.28 64 logging ld valid N N --- ---
11 706 tp-1-sd-0.28 70 stale ld valid N N --- ---
11 707 tp-1-sd-0.28 76 logging ld valid N N --- ---
11 708 tp-1-sd-0.28 82 stale ld valid N N --- ---
11 709 tp-1-sd-0.28 88 stale ld valid N N --- ---
11 710 tp-1-sd-0.28 94 stale ld valid N N --- ---
11 711 tp-1-sd-0.28 100 stale ld valid N N --- ---
11 712 tp-1-sd-0.28 106 logging ld valid N N --- ---
11 713 tp-1-sd-0.28 112 stale ld valid N N --- ---
11 714 tp-1-sd-0.28 118 stale ld valid N N --- ---
11 715 tp-1-sd-0.28 124 stale ld valid N N --- ---
11 716 tp-1-sd-0.28 130 normal ld valid N N --- ---
11 717 tp-1-sd-0.28 136 stale ld valid N N --- ---
11 718 tp-1-sd-0.28 142 stale ld valid N N --- ---
11 719 tp-1-sd-0.28 148 stale ld valid N N --- ---
11 720 tp-1-sd-0.28 154 stale ld valid N N --- ---
11 721 tp-1-sd-0.28 160 stale ld valid N N --- ---
11 722 tp-1-sd-0.28 166 stale ld valid N N --- ---
11 723 tp-1-sd-0.28 172 stale ld valid N N --- ---
11 724 tp-1-sd-0.28 178 normal ld valid N N --- ---
11 725 tp-1-sd-0.28 184 stale ld valid N N --- ---
11 726 tp-1-sd-0.28 190 normal ld valid N N --- ---
11 727 tp-1-sd-0.28 196 logging ld valid N N --- ---
11 728 tp-1-sd-0.28 202 stale ld valid N N --- ---
11 729 tp-1-sd-0.28 208 logging ld valid N N --- ---
11 730 tp-1-sd-0.28 214 logging ld valid N N --- ---
11 731 tp-1-sd-0.28 220 stale ld valid N N --- ---
11 732 tp-1-sd-0.28 226 stale ld valid N N --- ---
11 733 tp-1-sd-0.28 232 normal ld valid N N --- ---
11 734 tp-1-sd-0.28 244 stale ld valid N N --- ---
11 735 tp-1-sd-0.28 250 stale ld valid N N --- ---
11 736 tp-1-sd-0.0 292 stale ld valid N N --- ---
11 737 tp-1-sd-0.0 304 stale ld valid N N --- ---
--------------------------------------------------------------
Total chunklets: 283

Yeah, that's a lot ...

Are any of your volumes offline/stale (DataStore3PAR01 or DataStore3PAR02)?

What you would have to do to be able to dismiss the pd is to continue the "movech -perm -ovrd 11:..." Where "..." is Chnk from the showpdch output.

However I am a little concern about those chunklets that are stale.... could be that they're fixed when moved, but it could also be that the chunklet is stale because the LD is locked due to raid set failure.

Do you have more than one failed drive in your system?

Same result:

3PAR cli% movech -perm -ovrd 11:452
Are you sure you want to move the chunklets?
select q=quit y=yes n=no: y
Error: Could not find a relocation target for chunklet 11:452

3PAR cli% movech -perm -ovrd 11:453
Are you sure you want to move the chunklets?
select q=quit y=yes n=no: y
Error: Could not find a relocation target for chunklet 11:453

any more ideas?