Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX] duration: 1m0.033052521s FAIL (2025-07-26 23:03:37 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 639916s) [upload=1083]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Jul 26 23:02:38.009431979 (1h10m53.790365367s) [upload=1100]: Starting upload Jul 26 23:02:38.009529742 (1h10m53.79046312s) [upload=1100]: Done starting upload Jul 26 23:02:38.009693140 (1h10m53.790626551s) [upload=1100]: Uploading segments... Jul 26 23:02:38.009988782 (1h10m53.790922148s) [upload=1100]: Got next segment. Inline: false Jul 26 23:02:38.010070015 (1h10m53.7910034s) [upload=1100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:02:38.010077679 (1h10m53.791011064s) [upload=1100]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:02:38.044231887 (1h10m53.825165274s) [seg_pos={0 0}, upload=1100]: Begin upload segment... Jul 26 23:02:38.044288604 (1h10m53.825221986s) [seg_pos={0 0}, upload=1100]: Done begin upload segment. Jul 26 23:02:38.044317043 (1h10m53.82525043s) [noise=true , address=64.49.13.139:4478 , piece_id=3GN3HISCZ4EEIID2GIUWAWGWILQKIQEDN25T5OSFTFCE3RLWOBKA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.044360663 (1h10m53.825294057s) [noise=true , address=45.91.116.11:4121 , piece_id=P76COMVRFHS4DLD45V5HWKZRLJDKYPL3M2IAPKDXUT4X2F6KV3CQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.044838801 (1h10m53.825772189s) [noise=true , address=77.90.31.19:4859 , piece_id=OIKITA4V76KN3VP7KPWZYHJTRA66RFFV5HFRWGEW46YLWIKZTUIA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.044953426 (1h10m53.82588681s) [noise=true , address=79.127.219.47:4061 , piece_id=FR5RPPWQ74GS7GLFKSQMAQ4HB2UTI6ZNKKGEGYNITPEE35KUTALA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.045307782 (1h10m53.826241184s) [noise=true , address=79.127.205.245:4243 , piece_id=KIBMUQOZG6EJFBWRKC5ZAIUJCKZXDH5GA4OJIK6M2JVXX76XT5SA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.045577917 (1h10m53.826511306s) [noise=true , address=79.127.205.234:4615 , piece_id=6DYCMDDTBMYEWJ5DMSKR5OBKD4SU7ZSKE3O4T4HZ6QRUTFN3GVJA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.045795589 (1h10m53.826728975s) [noise=true , address=45.91.116.9:4277 , piece_id=QWAN66EE7R54K2TZYJRFBBP3WHH3CVYG4UTJ2BCDSVKXRFSXBM2A, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046160990 (1h10m53.827094381s) [noise=true , address=79.127.205.235:4685 , piece_id=D2WL6BO4WMWX2VRVQ2V6TQIQPRONVJCWXZ4SGAM6DQ4CU3R3QELA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046168935 (1h10m53.827102298s) [noise=true , address=79.127.220.101:4048 , piece_id=F2NCRBZL4AVI6QAS2OQ6DBBSBEFMBT4TARCTZWJOKINI65VTMZHQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046544662 (1h10m53.827478048s) [noise=true , address=77.90.31.15:4727 , piece_id=6E7RVTU3GRXXXIOJWRIXW36JASGTOOYJ7JTG5VUUGGNJTJQKIRHA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046638748 (1h10m53.82757213s) [noise=true , address=79.127.219.48:4090 , piece_id=CZCMAKDM3UMRVCH226TCKAMN5PKHH3KRWQPAENHTVKKU2AANJGAA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046971631 (1h10m53.827905015s) [noise=true , address=45.91.116.14:4206 , piece_id=GS53BPTRXHDKND5G5CI7QVK2LMOTA2AGSBI3RQSDKLMY5UYWWYFA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.047469479 (1h10m53.828402863s) [noise=true , address=79.127.219.38:4004 , piece_id=I4N3UNU4PMU6GRS3ZVO6FYYYXSLLXSI45GBXJN3KORIAIDLKNA2A, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.047886394 (1h10m53.828819784s) [noise=true , address=77.90.31.11:4571 , piece_id=QVQ45R3X7IMH22OC3MTV3MJC6YQSCVWNEEZB3266TPDSB2NDZAKQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.048312600 (1h10m53.829245988s) [noise=true , address=79.127.205.245:4224 , piece_id=3ST6YVMWDM2D45WIGTCONJH4MALS3STDK5BSHVPSCOFPKA65DHZQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.048741550 (1h10m53.829674942s) [noise=true , address=77.90.31.12:4608 , piece_id=5GNVVBTWM75DK7J75ARYEA7Y6HBCS5UJJJK6E3J6ZFRVAYBEFY3Q, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.048867751 (1h10m53.829801138s) [noise=true , address=79.127.201.215:4358 , piece_id=RRICN3R6RRF2MIBZM2ZXIXI6NG2QKRAAMCTZ5ZUQOQBRHHHWANDQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.049276904 (1h10m53.830210291s) [noise=true , address=79.127.219.51:4212 , piece_id=2V5P22DCZLVE5VSEKPISV2RZBFRBCUSFD3CG63IHKUZLIT56LORA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.113183749 (1h10m53.894117126s) [noise=true , address=79.127.219.51:4212 , piece_id=2V5P22DCZLVE5VSEKPISV2RZBFRBCUSFD3CG63IHKUZLIT56LORA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 23:02:38.135417853 (1h10m53.916351218s) [upload=1100]: Next returned nil segment Jul 26 23:02:38.135441335 (1h10m53.916374722s) [upload=1100]: Waiting for error group managing segments... Jul 26 23:02:38.296608477 (1h10m54.077541863s) [noise=true , address=79.127.219.38:4004 , piece_id=I4N3UNU4PMU6GRS3ZVO6FYYYXSLLXSI45GBXJN3KORIAIDLKNA2A, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.553667152 (1h10m54.334600522s) [noise=true , address=77.90.31.12:4608 , piece_id=5GNVVBTWM75DK7J75ARYEA7Y6HBCS5UJJJK6E3J6ZFRVAYBEFY3Q, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.631703695 (1h10m54.412637075s) [noise=true , address=79.127.205.235:4685 , piece_id=D2WL6BO4WMWX2VRVQ2V6TQIQPRONVJCWXZ4SGAM6DQ4CU3R3QELA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.646996395 (1h10m54.427929767s) [noise=true , address=79.127.205.245:4243 , piece_id=KIBMUQOZG6EJFBWRKC5ZAIUJCKZXDH5GA4OJIK6M2JVXX76XT5SA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.660623879 (1h10m54.441557271s) [noise=true , address=79.127.205.245:4224 , piece_id=3ST6YVMWDM2D45WIGTCONJH4MALS3STDK5BSHVPSCOFPKA65DHZQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.661444603 (1h10m54.442377975s) [noise=true , address=79.127.205.234:4615 , piece_id=6DYCMDDTBMYEWJ5DMSKR5OBKD4SU7ZSKE3O4T4HZ6QRUTFN3GVJA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:40.480167315 (1h10m56.261100702s) [noise=true , address=79.127.220.101:4048 , piece_id=F2NCRBZL4AVI6QAS2OQ6DBBSBEFMBT4TARCTZWJOKINI65VTMZHQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:40.846007988 (1h10m56.626941368s) [noise=true , address=79.127.219.48:4090 , piece_id=CZCMAKDM3UMRVCH226TCKAMN5PKHH3KRWQPAENHTVKKU2AANJGAA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:40.905994229 (1h10m56.686927607s) [noise=true , address=79.127.219.47:4061 , piece_id=FR5RPPWQ74GS7GLFKSQMAQ4HB2UTI6ZNKKGEGYNITPEE35KUTALA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:44.563162762 (1h11m0.344096151s) [noise=true , address=45.91.116.14:4206 , piece_id=GS53BPTRXHDKND5G5CI7QVK2LMOTA2AGSBI3RQSDKLMY5UYWWYFA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 23:02:45.387863794 (1h11m1.168797163s) [noise=true , address=45.91.116.9:4277 , piece_id=QWAN66EE7R54K2TZYJRFBBP3WHH3CVYG4UTJ2BCDSVKXRFSXBM2A, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:46.641102649 (1h11m2.422036011s) [noise=true , address=64.49.13.139:4478 , piece_id=3GN3HISCZ4EEIID2GIUWAWGWILQKIQEDN25T5OSFTFCE3RLWOBKA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:56.398068001 (1h11m12.179001368s) [noise=true , address=77.90.31.19:4859 , piece_id=OIKITA4V76KN3VP7KPWZYHJTRA66RFFV5HFRWGEW46YLWIKZTUIA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:03:12.478927638 (1h11m28.259861015s) [noise=true , address=45.91.116.11:4121 , piece_id=P76COMVRFHS4DLD45V5HWKZRLJDKYPL3M2IAPKDXUT4X2F6KV3CQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:1wkujwGr7G6yGYFLiYcxfBnN1pmKY6mHXoAzZWAjGCCiWWZNdh, address:45.91.116.11:4121): protocol: storage node overloaded, request limit: 2000 Jul 26 23:03:22.198648346 (1h11m37.979581727s) [noise=true , address=79.127.201.215:4358 , piece_id=RRICN3R6RRF2MIBZM2ZXIXI6NG2QKRAAMCTZ5ZUQOQBRHHHWANDQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:03:37.964568820 (1h11m53.7455022s) [noise=true , address=77.90.31.11:4571 , piece_id=QVQ45R3X7IMH22OC3MTV3MJC6YQSCVWNEEZB3266TPDSB2NDZAKQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: write tcp 159.223.143.39:42210->77.90.31.11:4571: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:03:37.965835006 (1h11m53.746768388s) [noise=true , address=77.90.31.15:4727 , piece_id=6E7RVTU3GRXXXIOJWRIXW36JASGTOOYJ7JTG5VUUGGNJTJQKIRHA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: write tcp 159.223.143.39:36482->77.90.31.15:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:03:37.967314376 (1h11m53.74824777s) [seg_pos={0 0}, upload=1100]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:03:37.994728808 (1h11m53.775662191s) [upload=1100]: Done uploading segments... Jul 26 23:03:37.994772690 (1h11m53.775706073s) [upload=1100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.033076235s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X........X................................................] duration: 402.013851ms FAIL (2025-07-26 23:03:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 402.051683ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 182.901955ms FAIL (2025-07-26 23:03:14 UTC) orbiter warning: uplink: too many requests Duration: 182.930296ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 203.446951ms FAIL (2025-07-26 23:02:51 UTC) orbiter warning: uplink: too many requests Duration: 203.519434ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.............X................X........................X..] duration: 1.290697241s FAIL (2025-07-26 23:02:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 1.29072857s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [X........X.X.X...XX.................X.X..........XX...X...X.] duration: 13.715032586s FAIL (2025-07-26 23:02:39 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.715054331s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXX] duration: 1m0.031115054s FAIL (2025-07-26 23:02:37 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 1083]: Done begin upload segment. Jul 26 23:01:37.955259301 (1h9m53.736192688s) [noise=true , address=77.90.31.10:4534 , piece_id=UG5WDILADUZ2YXHCPFJHLVFGJG7HGNIVWDNZRR6TQSYH6PUJJTCA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.955276692 (1h9m53.736210081s) [noise=true , address=77.90.31.16:4736 , piece_id=NHAJJTMG5PERTFZR6WD4YQD2ZQ5TECB3LPVGT76UXHDXU7KU2FPQ, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.955749101 (1h9m53.736682469s) [noise=true , address=77.90.31.11:4585 , piece_id=BXT66X4AXVPI5GUJCIJFLPG4WTTCUBZHL2ENNBY3AEF34CKNIONA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.955798941 (1h9m53.736732328s) [noise=true , address=79.127.205.233:4647 , piece_id=TA6V2BTP63SCNHEEHREUAUFSAZDAFK7PTHBDF356XUAEYOJSTVTQ, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.956178165 (1h9m53.737111553s) [noise=true , address=156.146.43.225:4434 , piece_id=7SCYRR6HLFEJ3CTGVHUAKUPHETXFN25JO5IB7UCFSKKSB2T2OPSQ, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.956375061 (1h9m53.737308441s) [noise=true , address=77.90.31.15:4729 , piece_id=NLOGILCQR5KWDQKLMMTD7SNJE554ETO6NY3FB6ZD24FLVQB2LPDQ, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.956592618 (1h9m53.737525985s) [noise=true , address=156.146.43.225:4449 , piece_id=XFJXEWDXZX6JBXWMCK6LNTFESOGRGNWIFSQ744BQOSF5XU4RC2FA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.956829052 (1h9m53.737762434s) [noise=true , address=79.127.205.245:4225 , piece_id=Q72CVB6AKSP2JZ4TNDM2ALOSJNN2I2MREPP7QCWOJRFKVGWMNSIA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.956975793 (1h9m53.737909194s) [noise=true , address=79.127.205.245:4241 , piece_id=R2C7HZUI6O2SSM6CRKML3CQG73BHISVD375VPG64MPWHY4NYIQRA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.957269512 (1h9m53.738202893s) [noise=true , address=77.90.31.17:4800 , piece_id=STTGDD2DRKHT4JQYSTJSJR6ZXWGK3HJK6OZWYVF5AJD6QDUSTPPA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.957405477 (1h9m53.738338868s) [noise=true , address=79.127.219.48:4090 , piece_id=IEQCATMUBUYU4APPUPSE6V4WQ4WETSL63HJUAQUDMX6PG7XXKHSA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.957675299 (1h9m53.738608682s) [noise=true , address=77.90.31.19:4871 , piece_id=IPV34RFOGY75HQ5DQPGGDAKXOZGAMOVFZO6KL72HB4I4MU3ZNXZA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.957750452 (1h9m53.738683852s) [noise=true , address=79.127.205.246:4278 , piece_id=4KZ3G6HYD22GA7B7TGTGDWQXONCIEZST7EH7MX4FR6ELBECA6TRQ, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.958075199 (1h9m53.739008578s) [noise=true , address=79.127.205.235:4690 , piece_id=7T5XI6OGOLG2NQWUUJZJPGLB4GXS475FFGK6BPUVMMNTPBEM2BZA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.958107878 (1h9m53.739041257s) [noise=true , address=79.127.205.234:4601 , piece_id=UNCAAY2JYM2KQ3ROLPPDNWC2UJ6J53LAOF3X67CO4ZH72VQWX3NQ, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.958492712 (1h9m53.73942609s) [noise=true , address=79.127.219.38:4009 , piece_id=TNR6B2SXPHPPXEIEGMDOCUZHAQSZW3QPI2DYSXOU67FW2SR7TGJA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.958944171 (1h9m53.739877554s) [noise=true , address=77.90.31.12:4611 , piece_id=RBE2BQV5TZY5R73MPIHPHCP7HK3KPKLF6VUT66VSQPXK6T4V3UAA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.959452651 (1h9m53.740386031s) [noise=true , address=77.90.31.11:4572 , piece_id=MLCI7LJHVLOQHRGBFDGJUSZULLTTELKN7R5OV5VXOTOBPJPR35OQ, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:38.005878674 (1h9m53.786812069s) [noise=true , address=77.90.31.19:4871 , piece_id=IPV34RFOGY75HQ5DQPGGDAKXOZGAMOVFZO6KL72HB4I4MU3ZNXZA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: storage node overloaded, request limit: 2000 Jul 26 23:01:38.026684894 (1h9m53.807618274s) [noise=true , address=79.127.219.48:4090 , piece_id=IEQCATMUBUYU4APPUPSE6V4WQ4WETSL63HJUAQUDMX6PG7XXKHSA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 23:01:38.055552473 (1h9m53.836485838s) [upload=1083]: Next returned nil segment Jul 26 23:01:38.055583547 (1h9m53.836516934s) [upload=1083]: Waiting for error group managing segments... Jul 26 23:01:38.147086153 (1h9m53.928019518s) [noise=true , address=79.127.219.38:4009 , piece_id=TNR6B2SXPHPPXEIEGMDOCUZHAQSZW3QPI2DYSXOU67FW2SR7TGJA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: <nil> Jul 26 23:01:38.441336814 (1h9m54.222270203s) [noise=true , address=156.146.43.225:4449 , piece_id=XFJXEWDXZX6JBXWMCK6LNTFESOGRGNWIFSQ744BQOSF5XU4RC2FA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 23:01:38.595971390 (1h9m54.376904755s) [noise=true , address=79.127.205.246:4278 , piece_id=4KZ3G6HYD22GA7B7TGTGDWQXONCIEZST7EH7MX4FR6ELBECA6TRQ, seg_pos={0 0}, upload=1083]: Done uploading piece. err: <nil> Jul 26 23:01:38.606153946 (1h9m54.38708731s) [noise=true , address=79.127.205.245:4225 , piece_id=Q72CVB6AKSP2JZ4TNDM2ALOSJNN2I2MREPP7QCWOJRFKVGWMNSIA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: <nil> Jul 26 23:01:38.624838426 (1h9m54.405771805s) [noise=true , address=79.127.205.235:4690 , piece_id=7T5XI6OGOLG2NQWUUJZJPGLB4GXS475FFGK6BPUVMMNTPBEM2BZA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: <nil> Jul 26 23:01:38.642426432 (1h9m54.423359804s) [noise=true , address=79.127.205.233:4647 , piece_id=TA6V2BTP63SCNHEEHREUAUFSAZDAFK7PTHBDF356XUAEYOJSTVTQ, seg_pos={0 0}, upload=1083]: Done uploading piece. err: <nil> Jul 26 23:01:38.657972779 (1h9m54.438906149s) [noise=true , address=79.127.205.234:4601 , piece_id=UNCAAY2JYM2KQ3ROLPPDNWC2UJ6J53LAOF3X67CO4ZH72VQWX3NQ, seg_pos={0 0}, upload=1083]: Done uploading piece. err: <nil> Jul 26 23:01:38.658056837 (1h9m54.438990215s) [noise=true , address=79.127.205.245:4241 , piece_id=R2C7HZUI6O2SSM6CRKML3CQG73BHISVD375VPG64MPWHY4NYIQRA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: <nil> Jul 26 23:01:40.030667657 (1h9m55.811601026s) [noise=true , address=77.90.31.12:4611 , piece_id=RBE2BQV5TZY5R73MPIHPHCP7HK3KPKLF6VUT66VSQPXK6T4V3UAA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: <nil> Jul 26 23:01:41.700806954 (1h9m57.481740327s) [noise=true , address=156.146.43.225:4434 , piece_id=7SCYRR6HLFEJ3CTGVHUAKUPHETXFN25JO5IB7UCFSKKSB2T2OPSQ, seg_pos={0 0}, upload=1083]: Done uploading piece. err: <nil> Jul 26 23:01:50.720954465 (1h10m6.501887844s) [noise=true , address=77.90.31.17:4800 , piece_id=STTGDD2DRKHT4JQYSTJSJR6ZXWGK3HJK6OZWYVF5AJD6QDUSTPPA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: ecclient: upload failed (node:1TxoPpye17ET1GR52xy2EmkUdq83x4T9TsfuwrKr8cYXf3HLYt, address:77.90.31.17:4800): protocol: storage node overloaded, request limit: 2000 Jul 26 23:02:37.865994557 (1h10m53.64692795s) [noise=true , address=77.90.31.11:4585 , piece_id=BXT66X4AXVPI5GUJCIJFLPG4WTTCUBZHL2ENNBY3AEF34CKNIONA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: rpc: dial tcp 77.90.31.11:4585: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:02:37.866189732 (1h10m53.647123117s) [noise=true , address=77.90.31.15:4729 , piece_id=NLOGILCQR5KWDQKLMMTD7SNJE554ETO6NY3FB6ZD24FLVQB2LPDQ, seg_pos={0 0}, upload=1083]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: rpc: dial tcp 77.90.31.15:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:02:37.866275913 (1h10m53.647209292s) [noise=true , address=77.90.31.11:4572 , piece_id=MLCI7LJHVLOQHRGBFDGJUSZULLTTELKN7R5OV5VXOTOBPJPR35OQ, seg_pos={0 0}, upload=1083]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: write tcp 159.223.143.39:32774->77.90.31.11:4572: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:02:37.866381094 (1h10m53.647314486s) [noise=true , address=77.90.31.16:4736 , piece_id=NHAJJTMG5PERTFZR6WD4YQD2ZQ5TECB3LPVGT76UXHDXU7KU2FPQ, seg_pos={0 0}, upload=1083]: Done uploading piece. err: ecclient: upload failed (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, address:77.90.31.16:4736): protocol: write tcp 159.223.143.39:50350->77.90.31.16:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:02:37.866439679 (1h10m53.647373062s) [noise=true , address=77.90.31.10:4534 , piece_id=UG5WDILADUZ2YXHCPFJHLVFGJG7HGNIVWDNZRR6TQSYH6PUJJTCA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: read tcp 159.223.143.39:41702->77.90.31.10:4534: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:02:37.866479299 (1h10m53.647412688s) [seg_pos={0 0}, upload=1083]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:02:37.892659719 (1h10m53.673593109s) [upload=1083]: Done uploading segments... Jul 26 23:02:37.892706533 (1h10m53.673639916s) [upload=1083]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.031143897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X........X...............................................] duration: 620.592561ms FAIL (2025-07-26 23:02:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 620.6215ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.X........X.X.X...XX.................X.X..........XX...X...X] duration: 1m0.000547833s FAIL (2025-07-26 23:02:21 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00058487s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.............X................X........................X.] duration: 466.03655ms FAIL (2025-07-26 23:02:13 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 466.067961ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 194.636927ms FAIL (2025-07-26 23:02:06 UTC) orbiter warning: uplink: too many requests Duration: 194.668572ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X........X..............................................] duration: 553.132114ms FAIL (2025-07-26 23:02:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 553.168331ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.............X................X........................X] duration: 30.03288738s FAIL (2025-07-26 23:01:50 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.03294042s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXX] duration: 1m0.027591794s FAIL (2025-07-26 23:01:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== s) [upload=1048]: Done uploading segments... Jul 26 23:00:28.609366837 (1h8m44.390300223s) [upload=1048]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Jul 26 23:00:28.779559960 (1h8m44.560493328s) [upload=1066]: Starting upload Jul 26 23:00:28.779653135 (1h8m44.560586515s) [upload=1066]: Done starting upload Jul 26 23:00:28.779750366 (1h8m44.560683746s) [upload=1066]: Uploading segments... Jul 26 23:00:28.779923494 (1h8m44.560856871s) [upload=1066]: Got next segment. Inline: false Jul 26 23:00:28.779977957 (1h8m44.560911319s) [upload=1066]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:00:28.779987601 (1h8m44.560921051s) [upload=1066]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:00:28.876762903 (1h8m44.657696341s) [seg_pos={0 0}, upload=1066]: Begin upload segment... Jul 26 23:00:28.876845194 (1h8m44.657778585s) [seg_pos={0 0}, upload=1066]: Done begin upload segment. Jul 26 23:00:28.876888922 (1h8m44.65782231s) [noise=true , address=77.90.31.19:4871 , piece_id=GF7VPASF5UYDVYIZPBYB6B3X7ZVT6NE5HBBEGBFZS2PC6I4BKY4A, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.877532636 (1h8m44.658466046s) [noise=true , address=79.127.205.234:4603 , piece_id=GMJX6KUOV4T6FLFJDBWOQAZKY4IRQBJAKIBI6RJHERJ7HB3LVG4Q, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.878032452 (1h8m44.658965841s) [noise=true , address=45.91.116.5:4310 , piece_id=4I3OEX4PZFA5S2WL2RQ3AQXB6SXVBBFPYWSWZ4GAQONNEENKPHYQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.878520889 (1h8m44.65945434s) [noise=true , address=77.90.31.12:4601 , piece_id=KB4MRV7ZA3HR2AFVCPLHVU3VCPUDKKOG6XZASYNKZGHQDD3HMTAQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.878798467 (1h8m44.659731852s) [noise=true , address=45.91.116.11:4121 , piece_id=PAE6BE67CPC5WSBULNMGXI3G254M43SQKTXWMYISJILZYTPKCOZQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.882673502 (1h8m44.663606868s) [noise=true , address=79.127.219.38:4006 , piece_id=QRZQFXG53KTASLNYDPJXAUE5RUM36AM4DSBFQZ33FIWLTZXIAQMQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.883549394 (1h8m44.664482782s) [noise=true , address=79.127.205.235:4657 , piece_id=YDWT72JHEPG6EGIG2HHTV4SSBDVWN6XMV2VBOZZ35MRXKHZ6VHZQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.884146773 (1h8m44.665080153s) [noise=true , address=79.127.219.38:4005 , piece_id=AHJN4QNVYHEH27T2BNWCLDUZ4NURHDPAVYMGXS275YLZZMNRYRYA, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.884904311 (1h8m44.665837698s) [noise=true , address=77.90.31.21:4897 , piece_id=77JABZES2DNPGERLELV6VYHSO3CENV4J6BF6WG35IKWOJ6S64PQQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.886476794 (1h8m44.667410183s) [noise=true , address=79.127.205.246:4278 , piece_id=BPCV4TRQBW26MCFOQGDOLS4RMG4GIYHUWNNW25GIQ7BMKDDHZUMQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.887056314 (1h8m44.667989703s) [noise=true , address=79.127.205.233:4647 , piece_id=OGOR3NQX7KGYY36IGCXDIW4HSEJYE6OQNAOOJOU3MUA2NUVYOWRQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.887597951 (1h8m44.66853134s) [noise=true , address=45.91.116.14:4194 , piece_id=MV65HCFWZZWTISYGQ3CN3MJW65PKTPHVKZMUZGTY5TTWYP6O7NTQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.888162033 (1h8m44.669095421s) [noise=true , address=45.91.116.14:4222 , piece_id=L6AUWAS3LFK45PVRLOQYAGE5A6TODMQI6CC4UET4KZH3BN72JQLA, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.888924172 (1h8m44.669857557s) [noise=true , address=45.91.116.13:4175 , piece_id=3E2T7QCWKLSKF35CUNKRQGOEE6TPB4M6SUJ6LL4BD7Q6SHCWH7SQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.889840118 (1h8m44.670773504s) [noise=true , address=79.127.219.48:4090 , piece_id=YSBAII4FSIBGT27EJUE6GYD44NVPM5QT4HIMRCMP2XD7Z2ZEESRQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.890474302 (1h8m44.671407703s) [noise=true , address=77.90.31.13:4644 , piece_id=OOABN4245V6AO2LIZFGMMPAWLZCY7GCGFNUO22QZI46RFGIVE4CA, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.891354729 (1h8m44.67228812s) [noise=true , address=64.49.13.170:4719 , piece_id=KITJUI5UO6AS4FPLDUUI7IBDSGLKMVUE7ASDEXWRI34U6GYP7NPQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.892054191 (1h8m44.67298758s) [noise=true , address=64.49.13.139:4478 , piece_id=MNQFARQXMP532LW3MXX7ORIQJAD7RFYD6TVTZ5QFNRMWMYECAYBQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.958309374 (1h8m44.739242742s) [noise=true , address=79.127.219.38:4006 , piece_id=QRZQFXG53KTASLNYDPJXAUE5RUM36AM4DSBFQZ33FIWLTZXIAQMQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: ecclient: upload failed (node:18qiLeQCqaa6Sa898QMfKEZo68t1JKbDMk4ixJjTzcFgL7aes3, address:79.127.219.38:4006): protocol: storage node overloaded, request limit: 2000 Jul 26 23:00:29.060212463 (1h8m44.841145833s) [upload=1066]: Next returned nil segment Jul 26 23:00:29.060380258 (1h8m44.841313646s) [upload=1066]: Waiting for error group managing segments... Jul 26 23:00:29.265127244 (1h8m45.046060624s) [noise=true , address=77.90.31.12:4601 , piece_id=KB4MRV7ZA3HR2AFVCPLHVU3VCPUDKKOG6XZASYNKZGHQDD3HMTAQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:29.574612772 (1h8m45.355546155s) [noise=true , address=79.127.205.233:4647 , piece_id=OGOR3NQX7KGYY36IGCXDIW4HSEJYE6OQNAOOJOU3MUA2NUVYOWRQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:29.592311070 (1h8m45.373244454s) [noise=true , address=79.127.205.246:4278 , piece_id=BPCV4TRQBW26MCFOQGDOLS4RMG4GIYHUWNNW25GIQ7BMKDDHZUMQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:29.603602389 (1h8m45.384535777s) [noise=true , address=79.127.205.234:4603 , piece_id=GMJX6KUOV4T6FLFJDBWOQAZKY4IRQBJAKIBI6RJHERJ7HB3LVG4Q, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:29.733428276 (1h8m45.514361647s) [noise=true , address=79.127.205.235:4657 , piece_id=YDWT72JHEPG6EGIG2HHTV4SSBDVWN6XMV2VBOZZ35MRXKHZ6VHZQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:30.299891512 (1h8m46.080824891s) [noise=true , address=45.91.116.14:4194 , piece_id=MV65HCFWZZWTISYGQ3CN3MJW65PKTPHVKZMUZGTY5TTWYP6O7NTQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:30.302944110 (1h8m46.083877493s) [noise=true , address=45.91.116.5:4310 , piece_id=4I3OEX4PZFA5S2WL2RQ3AQXB6SXVBBFPYWSWZ4GAQONNEENKPHYQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:30.487640710 (1h8m46.268574086s) [noise=true , address=64.49.13.170:4719 , piece_id=KITJUI5UO6AS4FPLDUUI7IBDSGLKMVUE7ASDEXWRI34U6GYP7NPQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:31.985030848 (1h8m47.765964225s) [noise=true , address=77.90.31.19:4871 , piece_id=GF7VPASF5UYDVYIZPBYB6B3X7ZVT6NE5HBBEGBFZS2PC6I4BKY4A, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:32.180639051 (1h8m47.961572424s) [noise=true , address=45.91.116.11:4121 , piece_id=PAE6BE67CPC5WSBULNMGXI3G254M43SQKTXWMYISJILZYTPKCOZQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:35.193917160 (1h8m50.974850528s) [noise=true , address=79.127.219.48:4090 , piece_id=YSBAII4FSIBGT27EJUE6GYD44NVPM5QT4HIMRCMP2XD7Z2ZEESRQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:37.970649671 (1h8m53.751583037s) [noise=true , address=79.127.219.38:4005 , piece_id=AHJN4QNVYHEH27T2BNWCLDUZ4NURHDPAVYMGXS275YLZZMNRYRYA, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:37.973852054 (1h8m53.754785434s) [noise=true , address=45.91.116.13:4175 , piece_id=3E2T7QCWKLSKF35CUNKRQGOEE6TPB4M6SUJ6LL4BD7Q6SHCWH7SQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: ecclient: upload failed (node:124YLi7uuYNfk8TYrgdAkmrGnKmCfRC1MBmeJypA2hwHyWdp43U, address:45.91.116.13:4175): protocol: storage node overloaded, request limit: 2000 Jul 26 23:00:42.795986444 (1h8m58.576919825s) [noise=true , address=45.91.116.14:4222 , piece_id=L6AUWAS3LFK45PVRLOQYAGE5A6TODMQI6CC4UET4KZH3BN72JQLA, seg_pos={0 0}, upload=1066]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 23:00:47.116949998 (1h9m2.897883378s) [noise=true , address=64.49.13.139:4478 , piece_id=MNQFARQXMP532LW3MXX7ORIQJAD7RFYD6TVTZ5QFNRMWMYECAYBQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:01:28.707826424 (1h9m44.488759804s) [noise=true , address=77.90.31.13:4644 , piece_id=OOABN4245V6AO2LIZFGMMPAWLZCY7GCGFNUO22QZI46RFGIVE4CA, seg_pos={0 0}, upload=1066]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: rpc: dial tcp 77.90.31.13:4644: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:28.707936095 (1h9m44.488869474s) [noise=true , address=77.90.31.21:4897 , piece_id=77JABZES2DNPGERLELV6VYHSO3CENV4J6BF6WG35IKWOJ6S64PQQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: ecclient: upload failed (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x, address:77.90.31.21:4897): protocol: write tcp 159.223.143.39:38006->77.90.31.21:4897: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:28.707958133 (1h9m44.48889151s) [seg_pos={0 0}, upload=1066]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:01:28.733358789 (1h9m44.514292169s) [upload=1066]: Done uploading segments... Jul 26 23:01:28.733388630 (1h9m44.514322012s) [upload=1066]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.027630659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X....X........X.............................................] duration: 385.283083ms FAIL (2025-07-26 23:01:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 385.314746ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 193.425992ms FAIL (2025-07-26 23:01:00 UTC) orbiter warning: uplink: too many requests Duration: 193.595142ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..X........X.X.X...XX.................X.X..........XX...X...] duration: 24.171562333s FAIL (2025-07-26 23:00:50 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.171628227s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XX] duration: 1m0.026231565s FAIL (2025-07-26 23:00:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Jul 26 22:59:28.632785432 (1h7m44.413718797s) [upload=1048]: Starting upload Jul 26 22:59:28.632852774 (1h7m44.413786151s) [upload=1048]: Done starting upload Jul 26 22:59:28.632966996 (1h7m44.413900373s) [upload=1048]: Uploading segments... Jul 26 22:59:28.633135249 (1h7m44.414068619s) [upload=1048]: Got next segment. Inline: false Jul 26 22:59:28.633179735 (1h7m44.414113103s) [upload=1048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:59:28.633185044 (1h7m44.414118415s) [upload=1048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:59:28.675174091 (1h7m44.456107478s) [seg_pos={0 0}, upload=1048]: Begin upload segment... Jul 26 22:59:28.675319246 (1h7m44.456252633s) [noise=true , address=79.127.205.233:4640 , piece_id=M33KSG4V3CPE3DDPORB353USNKP5K2GZK23JQVINF3QAVLP42RHA, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.675985706 (1h7m44.456919104s) [noise=true , address=79.127.205.246:4273 , piece_id=EYG55YHPP4INLILEI53J4D76VE4DIGNUVHW46YKOJTH23Z2VRT2A, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.676703310 (1h7m44.457636714s) [noise=true , address=77.90.31.17:4800 , piece_id=DJ4TGCEYHNLBLLDMKHXQOTKGWHAC5GVQCRF6FJFAQW5JQQSC5JQQ, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.677403064 (1h7m44.458336451s) [noise=true , address=77.90.31.12:4608 , piece_id=D75VON4TP5PK24242JAXN33CEXTUP5GAYCCX236H5TCZENX6WWWA, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.677889334 (1h7m44.458822723s) [noise=true , address=77.90.31.11:4574 , piece_id=5S3J6PA7L5SCUVFX7MEK7APPDK6O4C547VU6TUGOS4QR3DSMFYMA, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.679977009 (1h7m44.460910389s) [seg_pos={0 0}, upload=1048]: Done begin upload segment. Jul 26 22:59:28.679998989 (1h7m44.460932355s) [noise=true , address=79.127.219.38:4006 , piece_id=R2Z4K2KZF64ASWD7XGF4GGCR5PM65KYKW64NV73ZBXIHDN7MFNVA, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.680624581 (1h7m44.461557974s) [noise=true , address=79.127.205.245:4236 , piece_id=2ELWIPVPX4FHJMHUVC4E2FGME6CJGUQZXRNQYH4IKM7F7ZGKUB6Q, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.681491378 (1h7m44.462424741s) [noise=true , address=64.49.13.140:4514 , piece_id=QNOHYVKSOXOXFLEKZFZDQ3EUMZTOJ5YSLRKS5UXLAUJELKWMAMSA, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.682360718 (1h7m44.463294106s) [noise=true , address=79.127.205.246:4262 , piece_id=4WJH3ALU6LIKR67AZOC6HH43VYRJCXNIYIJS3T6KFH4M7XFGHMQA, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.684909750 (1h7m44.465843119s) [noise=true , address=79.127.219.47:4048 , piece_id=NBYL4UPTLCLBB47SIHTEQ4PIM4K3SOA64RYEPM56VTT72QRLRTNQ, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.687832780 (1h7m44.468766169s) [noise=true , address=79.127.205.245:4243 , piece_id=GGG2MCOQXAW3FYALXOTKL4AJJ2QCJ6LYRKCQM2UIXF5MORJCKIAA, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.692412427 (1h7m44.473345791s) [noise=true , address=77.90.31.10:4538 , piece_id=5ZX4B5Z7LDJY3SMCGHBJ5RSHF26FGFPU6JV7IJU72HG4EIC3433Q, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.695376024 (1h7m44.476309418s) [noise=true , address=79.127.205.235:4679 , piece_id=JUUD3SGKUG2IF3VDOUFRTD4Z7AUJ5K3GXDOIL7X4ULSI5SJG63QA, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.700353033 (1h7m44.481286417s) [noise=true , address=79.127.205.235:4657 , piece_id=QVDH35DXHQLCPTB3D4ER4IW755UKEDBPSV6IEO3RWDVNCBCJH54A, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.702827442 (1h7m44.483760832s) [noise=true , address=77.90.31.10:4531 , piece_id=AAXYJAOQQP6KDLXKPPKTCKBWS5NBD7QUC4F6ONAXCJGMJN5UI7QQ, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.704352429 (1h7m44.485285814s) [noise=true , address=45.91.116.14:4194 , piece_id=VR5IKMXYQDAERLGFQWLHU36SN32RTOJMECAYUWMRSEUOPFRMGL2A, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.705071967 (1h7m44.486005351s) [noise=true , address=77.90.31.19:4867 , piece_id=HIONQWFHD6XCEJ4M2HGAYATF7JYTX3SD4DO3LACZE4FEFKQZ2XKQ, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.707684100 (1h7m44.488617489s) [noise=true , address=79.127.220.97:4361 , piece_id=LQH5P74Q2FFC66W4VVRDK5ULWYF3ALKD6DDQ46CFGN2MBKMTKRQA, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.720993283 (1h7m44.501926666s) [noise=true , address=79.127.219.47:4048 , piece_id=NBYL4UPTLCLBB47SIHTEQ4PIM4K3SOA64RYEPM56VTT72QRLRTNQ, seg_pos={0 0}, upload=1048]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:59:28.731535325 (1h7m44.512468692s) [noise=true , address=79.127.219.38:4006 , piece_id=R2Z4K2KZF64ASWD7XGF4GGCR5PM65KYKW64NV73ZBXIHDN7MFNVA, seg_pos={0 0}, upload=1048]: Done uploading piece. err: ecclient: upload failed (node:18qiLeQCqaa6Sa898QMfKEZo68t1JKbDMk4ixJjTzcFgL7aes3, address:79.127.219.38:4006): protocol: storage node overloaded, request limit: 2000 Jul 26 22:59:28.732993931 (1h7m44.513927315s) [noise=true , address=77.90.31.10:4531 , piece_id=AAXYJAOQQP6KDLXKPPKTCKBWS5NBD7QUC4F6ONAXCJGMJN5UI7QQ, seg_pos={0 0}, upload=1048]: Done uploading piece. err: ecclient: upload failed (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt, address:77.90.31.10:4531): protocol: storage node overloaded, request limit: 2000 Jul 26 22:59:28.759175430 (1h7m44.540108813s) [upload=1048]: Next returned nil segment Jul 26 22:59:28.759197828 (1h7m44.540131215s) [upload=1048]: Waiting for error group managing segments... Jul 26 22:59:28.773102137 (1h7m44.554035507s) [noise=true , address=77.90.31.12:4608 , piece_id=D75VON4TP5PK24242JAXN33CEXTUP5GAYCCX236H5TCZENX6WWWA, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:29.087575082 (1h7m44.868508449s) [noise=true , address=79.127.220.97:4361 , piece_id=LQH5P74Q2FFC66W4VVRDK5ULWYF3ALKD6DDQ46CFGN2MBKMTKRQA, seg_pos={0 0}, upload=1048]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 26 22:59:29.157137363 (1h7m44.938070756s) [noise=true , address=64.49.13.140:4514 , piece_id=QNOHYVKSOXOXFLEKZFZDQ3EUMZTOJ5YSLRKS5UXLAUJELKWMAMSA, seg_pos={0 0}, upload=1048]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 26 22:59:29.276319050 (1h7m45.057252413s) [noise=true , address=79.127.205.233:4640 , piece_id=M33KSG4V3CPE3DDPORB353USNKP5K2GZK23JQVINF3QAVLP42RHA, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:29.294250921 (1h7m45.075184291s) [noise=true , address=79.127.205.235:4679 , piece_id=JUUD3SGKUG2IF3VDOUFRTD4Z7AUJ5K3GXDOIL7X4ULSI5SJG63QA, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:29.304970209 (1h7m45.085903583s) [noise=true , address=79.127.205.245:4243 , piece_id=GGG2MCOQXAW3FYALXOTKL4AJJ2QCJ6LYRKCQM2UIXF5MORJCKIAA, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:29.306288359 (1h7m45.087221736s) [noise=true , address=79.127.205.246:4262 , piece_id=4WJH3ALU6LIKR67AZOC6HH43VYRJCXNIYIJS3T6KFH4M7XFGHMQA, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:29.334484858 (1h7m45.115418228s) [noise=true , address=79.127.205.246:4273 , piece_id=EYG55YHPP4INLILEI53J4D76VE4DIGNUVHW46YKOJTH23Z2VRT2A, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:29.342984491 (1h7m45.123917861s) [noise=true , address=79.127.205.245:4236 , piece_id=2ELWIPVPX4FHJMHUVC4E2FGME6CJGUQZXRNQYH4IKM7F7ZGKUB6Q, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:29.438509939 (1h7m45.219443316s) [noise=true , address=79.127.205.235:4657 , piece_id=QVDH35DXHQLCPTB3D4ER4IW755UKEDBPSV6IEO3RWDVNCBCJH54A, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:29.797630748 (1h7m45.578564125s) [noise=true , address=77.90.31.17:4800 , piece_id=DJ4TGCEYHNLBLLDMKHXQOTKGWHAC5GVQCRF6FJFAQW5JQQSC5JQQ, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:29.956952019 (1h7m45.73788539s) [noise=true , address=77.90.31.10:4538 , piece_id=5ZX4B5Z7LDJY3SMCGHBJ5RSHF26FGFPU6JV7IJU72HG4EIC3433Q, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:32.829679251 (1h7m48.61061262s) [noise=true , address=45.91.116.14:4194 , piece_id=VR5IKMXYQDAERLGFQWLHU36SN32RTOJMECAYUWMRSEUOPFRMGL2A, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 23:00:14.856239221 (1h8m30.637172585s) [noise=true , address=77.90.31.19:4867 , piece_id=HIONQWFHD6XCEJ4M2HGAYATF7JYTX3SD4DO3LACZE4FEFKQZ2XKQ, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 23:00:28.586262741 (1h8m44.367196115s) [noise=true , address=77.90.31.11:4574 , piece_id=5S3J6PA7L5SCUVFX7MEK7APPDK6O4C547VU6TUGOS4QR3DSMFYMA, seg_pos={0 0}, upload=1048]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 159.223.143.39:44748->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:28.586295677 (1h8m44.367229066s) [seg_pos={0 0}, upload=1048]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:00:28.609336109 (1h8m44.390269492s) [upload=1048]: Done uploading segments... Jul 26 23:00:28.609366837 (1h8m44.390300223s) [upload=1048]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.026255195s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X....X........X............................................] duration: 634.099618ms FAIL (2025-07-26 23:00:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 634.203413ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [...X........X.X.X...XX.................X.X..........XX...X..] duration: 11.755287686s FAIL (2025-07-26 23:00:17 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.755314408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.............X................X........................] duration: 781.075341ms FAIL (2025-07-26 23:00:10 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 781.113459ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [....X........X.X.X...XX.................X.X..........XX...X.] duration: 7.129169372s FAIL (2025-07-26 22:59:57 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.129185492s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.....X........X.X.X...XX.................X.X..........XX...X] duration: 1m0.000651635s FAIL (2025-07-26 22:59:50 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000688476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 187.435614ms FAIL (2025-07-26 22:59:49 UTC) orbiter warning: uplink: too many requests Duration: 187.476739ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X....X........X...........................................] duration: 1.757399725s FAIL (2025-07-26 22:59:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.75743928s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.............X................X.......................] duration: 420.863457ms FAIL (2025-07-26 22:59:29 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 420.89406ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.X] duration: 1m0.034150021s FAIL (2025-07-26 22:59:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 88 (1h6m7.548390157s) [upload=1022]: Done uploading segments... Jul 26 22:57:51.767478006 (1h6m7.548411369s) [upload=1022]: Upload finished. err: <nil> Jul 26 22:58:28.502897453 (1h6m44.283830836s) [upload=1033]: Starting upload Jul 26 22:58:28.502997665 (1h6m44.283931137s) [upload=1033]: Done starting upload Jul 26 22:58:28.503232307 (1h6m44.284165689s) [upload=1033]: Uploading segments... Jul 26 22:58:28.503465067 (1h6m44.284398456s) [upload=1033]: Got next segment. Inline: false Jul 26 22:58:28.503615098 (1h6m44.284548479s) [upload=1033]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:58:28.503623431 (1h6m44.284556812s) [upload=1033]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:58:28.570016233 (1h6m44.3509496s) [seg_pos={0 0}, upload=1033]: Begin upload segment... Jul 26 22:58:28.570079800 (1h6m44.351013168s) [seg_pos={0 0}, upload=1033]: Done begin upload segment. Jul 26 22:58:28.570123784 (1h6m44.351057162s) [noise=true , address=77.90.31.13:4629 , piece_id=XK2HCP5FIW3T47WV5RHHP7BSXVYULNMTP6A2KSVKJTZNRM5KXLVQ, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.570677324 (1h6m44.351610711s) [noise=true , address=79.127.220.97:4361 , piece_id=GMU7F6FUM7YIQBBI6LG5YBQSJY42GKQ5N6FEIY4QPQLKBZYZCH5Q, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.571171134 (1h6m44.352104532s) [noise=true , address=45.91.116.9:4273 , piece_id=6JQNZBCWTDTSHXPOXRLXKFBFKQ4SQLRPRFCOGAAURAR4ISMKEL7Q, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.571685876 (1h6m44.352619264s) [noise=true , address=79.127.201.215:4359 , piece_id=YQ3AWQ2MRCTRCPUHB2MWVXBYDZP5EIUAJ3UDKZO5J5VREGWJBA5Q, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.572137096 (1h6m44.353070481s) [noise=true , address=77.90.31.10:4540 , piece_id=5A4ZJZGTMUBPTZE6DF7M4SSUOBRCVZPZIC5UFUTGIBNXFSBHUO4Q, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.572549346 (1h6m44.353482732s) [noise=true , address=79.127.219.47:4048 , piece_id=MSZTOXCHWLTLROZB5QS2WNQ7XYXDP5GPYVCLV3RMLNQVHUALUHYA, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.572887011 (1h6m44.353820389s) [noise=true , address=79.127.219.38:4007 , piece_id=JACIQ7OSSWTUGPGIW3BMVXWGT3AFTLEVBNQM73N4JEXVNXCREFCQ, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.573219801 (1h6m44.354153185s) [noise=true , address=79.127.205.234:4616 , piece_id=YXP7BI3WRWSMIUZ32FZO25FMVIAIENXHNYWOCJD3KV4I3JSTKZSQ, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.573531179 (1h6m44.35446456s) [noise=true , address=64.49.13.140:4514 , piece_id=EBGC2D4HQK6JYC23HROMOTJWQDZCDHIJPR2BMQJ744SKYNL3CCPA, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.573886217 (1h6m44.354819599s) [noise=true , address=79.127.205.235:4685 , piece_id=NK6LAY2CEHNFUCZWTIVNJ5XVXCXNS677L76M6PEVACPC2MSUKZYQ, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.574311436 (1h6m44.355244833s) [noise=true , address=79.127.205.246:4276 , piece_id=355QNCO2HK7RNGGXRXPMU6TM6NNXKC3DJFULNLD7PIABLIYEDXFA, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.574807278 (1h6m44.355740672s) [noise=true , address=79.127.219.51:4212 , piece_id=VRTOZIPSPZOCY3POGZKE5NHFY3FP64WKKTS6LKUZKJVMENS7JPPQ, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.575296256 (1h6m44.356229641s) [noise=true , address=77.90.31.16:4767 , piece_id=YWOWZMTNGLCPX7BK4JFGWFOPE5MI73GZCKIVSU4Y7RJ2XCZ2HC3Q, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.575680350 (1h6m44.356613733s) [noise=true , address=79.127.219.48:4090 , piece_id=MN23T27ZNRREHVZKBPQTHEJTJFCPRIJV4EPSJQ6AW3HX6RQ67IIA, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.576113708 (1h6m44.357047095s) [noise=true , address=79.127.205.245:4241 , piece_id=6MM7WUJU7UXSFBAMBOMBJRGYPZ6XBFEPKKLQZOGGMQAG3VJARRNA, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.576504313 (1h6m44.357437692s) [noise=true , address=79.127.219.38:4018 , piece_id=HI24WLXWZU3SWVHVGYZKZERFEYLZHPESL6Z4YNPCRMRYK3NOJFJA, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.576915348 (1h6m44.357848728s) [noise=true , address=79.127.205.233:4640 , piece_id=232XIHDK3YILEYWSS2KYCSIX6ZHLM6C6SMSFN2DJN65LOR6JOSYA, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.577280068 (1h6m44.358213449s) [noise=true , address=45.91.116.14:4222 , piece_id=CWQTATXU2KKEDLGEVUE7M6QGF436CASNMATUS45VTOLAWP6GQZPQ, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.623880246 (1h6m44.404813609s) [noise=true , address=77.90.31.10:4540 , piece_id=5A4ZJZGTMUBPTZE6DF7M4SSUOBRCVZPZIC5UFUTGIBNXFSBHUO4Q, seg_pos={0 0}, upload=1033]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: storage node overloaded, request limit: 2000 Jul 26 22:58:28.632243188 (1h6m44.413176566s) [noise=true , address=79.127.219.47:4048 , piece_id=MSZTOXCHWLTLROZB5QS2WNQ7XYXDP5GPYVCLV3RMLNQVHUALUHYA, seg_pos={0 0}, upload=1033]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:58:28.671876505 (1h6m44.452809895s) [upload=1033]: Next returned nil segment Jul 26 22:58:28.671895946 (1h6m44.452829329s) [upload=1033]: Waiting for error group managing segments... Jul 26 22:58:28.816811973 (1h6m44.597745342s) [noise=true , address=45.91.116.14:4222 , piece_id=CWQTATXU2KKEDLGEVUE7M6QGF436CASNMATUS45VTOLAWP6GQZPQ, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:29.195730664 (1h6m44.976664033s) [noise=true , address=79.127.205.233:4640 , piece_id=232XIHDK3YILEYWSS2KYCSIX6ZHLM6C6SMSFN2DJN65LOR6JOSYA, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:29.196412264 (1h6m44.977345632s) [noise=true , address=79.127.205.246:4276 , piece_id=355QNCO2HK7RNGGXRXPMU6TM6NNXKC3DJFULNLD7PIABLIYEDXFA, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:29.196945070 (1h6m44.977878438s) [noise=true , address=79.127.205.234:4616 , piece_id=YXP7BI3WRWSMIUZ32FZO25FMVIAIENXHNYWOCJD3KV4I3JSTKZSQ, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:29.220488705 (1h6m45.00142208s) [noise=true , address=79.127.205.245:4241 , piece_id=6MM7WUJU7UXSFBAMBOMBJRGYPZ6XBFEPKKLQZOGGMQAG3VJARRNA, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:29.387673191 (1h6m45.16860656s) [noise=true , address=79.127.205.235:4685 , piece_id=NK6LAY2CEHNFUCZWTIVNJ5XVXCXNS677L76M6PEVACPC2MSUKZYQ, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:29.703206499 (1h6m45.484139875s) [noise=true , address=79.127.219.38:4018 , piece_id=HI24WLXWZU3SWVHVGYZKZERFEYLZHPESL6Z4YNPCRMRYK3NOJFJA, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:30.494455980 (1h6m46.275389348s) [noise=true , address=79.127.220.97:4361 , piece_id=GMU7F6FUM7YIQBBI6LG5YBQSJY42GKQ5N6FEIY4QPQLKBZYZCH5Q, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:30.666391987 (1h6m46.447325357s) [noise=true , address=79.127.219.51:4212 , piece_id=VRTOZIPSPZOCY3POGZKE5NHFY3FP64WKKTS6LKUZKJVMENS7JPPQ, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:30.802615577 (1h6m46.583548945s) [noise=true , address=79.127.219.38:4007 , piece_id=JACIQ7OSSWTUGPGIW3BMVXWGT3AFTLEVBNQM73N4JEXVNXCREFCQ, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:31.056423266 (1h6m46.837356644s) [noise=true , address=79.127.219.48:4090 , piece_id=MN23T27ZNRREHVZKBPQTHEJTJFCPRIJV4EPSJQ6AW3HX6RQ67IIA, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:31.338443482 (1h6m47.119376859s) [noise=true , address=64.49.13.140:4514 , piece_id=EBGC2D4HQK6JYC23HROMOTJWQDZCDHIJPR2BMQJ744SKYNL3CCPA, seg_pos={0 0}, upload=1033]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 26 22:58:36.604546701 (1h6m52.385480072s) [noise=true , address=79.127.201.215:4359 , piece_id=YQ3AWQ2MRCTRCPUHB2MWVXBYDZP5EIUAJ3UDKZO5J5VREGWJBA5Q, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:36.698911243 (1h6m52.47984461s) [noise=true , address=45.91.116.9:4273 , piece_id=6JQNZBCWTDTSHXPOXRLXKFBFKQ4SQLRPRFCOGAAURAR4ISMKEL7Q, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:59:28.471592863 (1h7m44.252526228s) [noise=true , address=77.90.31.13:4629 , piece_id=XK2HCP5FIW3T47WV5RHHP7BSXVYULNMTP6A2KSVKJTZNRM5KXLVQ, seg_pos={0 0}, upload=1033]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: write tcp 159.223.143.39:38664->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:59:28.471647640 (1h7m44.252581009s) [noise=true , address=77.90.31.16:4767 , piece_id=YWOWZMTNGLCPX7BK4JFGWFOPE5MI73GZCKIVSU4Y7RJ2XCZ2HC3Q, seg_pos={0 0}, upload=1033]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: write tcp 159.223.143.39:38400->77.90.31.16:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:59:28.471669232 (1h7m44.252602609s) [seg_pos={0 0}, upload=1033]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:59:28.501757396 (1h7m44.282690774s) [upload=1033]: Done uploading segments... Jul 26 22:59:28.501797672 (1h7m44.282731048s) [upload=1033]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.034196505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 185.258628ms FAIL (2025-07-26 22:59:17 UTC) orbiter warning: uplink: too many requests Duration: 185.286159ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.............X................X......................] duration: 1.837879948s FAIL (2025-07-26 22:59:16 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.837909093s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 219.930396ms FAIL (2025-07-26 22:58:58 UTC) orbiter warning: uplink: too many requests Duration: 219.975415ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X....X........X..........................................] duration: 416.964025ms FAIL (2025-07-26 22:58:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 416.984116ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [X......X.............X................X.....................] duration: 1.918305719s FAIL (2025-07-26 22:58:35 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.918337159s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 197.956293ms FAIL (2025-07-26 22:58:22 UTC) orbiter warning: uplink: too many requests Duration: 197.98939ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 195.120013ms FAIL (2025-07-26 22:57:59 UTC) orbiter warning: uplink: too many requests Duration: 195.150524ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [......X........X.X.X...XX.................X.X..........XX...] duration: 11.161040094s FAIL (2025-07-26 22:57:57 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.161059266s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.] duration: 5.782457085s FAIL (2025-07-26 22:57:53 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== .505442529s) [upload=1003]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:56:32.761319638 (1h4m48.542253027s) [seg_pos={0 0}, upload=1003]: Begin upload segment... Jul 26 22:56:32.761432974 (1h4m48.542366355s) [seg_pos={0 0}, upload=1003]: Done begin upload segment. Jul 26 22:56:32.761477820 (1h4m48.542411202s) [noise=true , address=79.127.220.101:4048 , piece_id=TVP345FFF2K6UEF2H22BEVWU3O5WH3XDGTKDTUDKYD67OG7O67QA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.762109645 (1h4m48.543043034s) [noise=true , address=77.90.31.16:4746 , piece_id=WMKPRDVL2SEZJ3UJEQHBIH2JS2JY66ZNUH2XGFRZQDCM2T6OYHWQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.762361611 (1h4m48.543294999s) [noise=true , address=64.49.13.170:4719 , piece_id=LCPCJJ7AYZKDGP5JAY4MLJEJO6ZJJHXXLD3Q3KTSSIEMLDLHUARA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.762780632 (1h4m48.543714033s) [noise=true , address=77.90.31.17:4782 , piece_id=5XFMV3EQHUWELC6M7MZQHA7ETAJI3EI6FOZGJ4M5D3DGZ2YV7LFQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.762908017 (1h4m48.543841405s) [noise=true , address=79.127.201.215:4350 , piece_id=6K564YWYL4DCFPKMIDIOI7ND5OPD2EJCDTMIBNLBHZQXDWIRW2PQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.763409578 (1h4m48.544342966s) [noise=true , address=64.49.13.169:4716 , piece_id=EIOMDZATNT7APLWUNT2VBA76EUNPT477USSBDDT7HECUQRD6BPMQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.763531573 (1h4m48.544464957s) [noise=true , address=79.127.205.246:4275 , piece_id=ZOLLIB2TPN46WHUMTOWKQL6UGXEC5MESP7KKHCTLOJPAAB5MGT3A, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.763877060 (1h4m48.544810515s) [noise=true , address=156.146.43.225:4448 , piece_id=DYKB22KF4QIW3RDZJCDZMXMZ3TV6DPYZFPTUBBNJ3H6GXD3XA66A, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.764091544 (1h4m48.545024933s) [noise=true , address=45.91.116.9:4288 , piece_id=TA7RSWL4GOEOZ3JWXCE62BKOZLMRHLGKN5SGWT4LC3PTZSLXE5NQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.764412768 (1h4m48.545346156s) [noise=true , address=79.127.201.215:4345 , piece_id=QBREW3FB467WCTFL5MB6DPWQXHZVUGBE2PV2LQYZU47AO7B3UTPA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.764678280 (1h4m48.54561167s) [noise=true , address=45.91.116.14:4222 , piece_id=BOBNQUQNXQCJRZWZAPQ5GCLQO73Y4437QM7JY66RIU76TBV6ZCNA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.764860900 (1h4m48.545794285s) [noise=true , address=45.91.116.9:4277 , piece_id=ZRAUWEPI7RWSZ27LW4IEEGLCWQPVPKBUIY24DUDUTYE6VSJUA24Q, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.765166007 (1h4m48.546099397s) [noise=true , address=77.90.31.10:4540 , piece_id=ZP3GZOMNTLQ23WNDVUTDRXCLWZ6WTJLBEVBA4C7OW5N37KCFHKDA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.765283087 (1h4m48.546216475s) [noise=true , address=79.127.205.245:4251 , piece_id=NERWPNU2KEMQZNPDZLSYNNX2J25H4S4HT4ZAPZW652QYMQHJ2WBQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.765695313 (1h4m48.546628702s) [noise=true , address=79.127.219.51:4210 , piece_id=E3O7WN4MJXAPYTJ3JBS2QB5DPIEFN2SBPKSWZQDXLP2DNY4MDLDA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.766142042 (1h4m48.547075431s) [noise=true , address=45.91.116.5:4310 , piece_id=YZ6ULUMBX7YOLR6KVMG5I4RQXGPL3USYJ547GCJEYIWJX6DKEIKA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.766570851 (1h4m48.54750424s) [noise=true , address=77.90.31.11:4581 , piece_id=6TPSO2XZ7GPQ46ZMIEUXITWLMQAMYNOYDB3XWUEX6LLUREBCUULA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.766997991 (1h4m48.547931379s) [noise=true , address=45.91.116.11:4151 , piece_id=5MV24DI4E3PR4SFDP5RIK4L3OLBZQC635TU3QHPWDLU3JAKO3XNA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.812919497 (1h4m48.593852878s) [noise=true , address=77.90.31.10:4540 , piece_id=ZP3GZOMNTLQ23WNDVUTDRXCLWZ6WTJLBEVBA4C7OW5N37KCFHKDA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:32.823436329 (1h4m48.604369716s) [noise=true , address=45.91.116.9:4277 , piece_id=ZRAUWEPI7RWSZ27LW4IEEGLCWQPVPKBUIY24DUDUTYE6VSJUA24Q, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:32.855001154 (1h4m48.635934542s) [upload=1003]: Next returned nil segment Jul 26 22:56:32.855041745 (1h4m48.635975129s) [upload=1003]: Waiting for error group managing segments... Jul 26 22:56:32.915064007 (1h4m48.695997397s) [noise=true , address=77.90.31.11:4581 , piece_id=6TPSO2XZ7GPQ46ZMIEUXITWLMQAMYNOYDB3XWUEX6LLUREBCUULA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:32.975276655 (1h4m48.756210041s) [noise=true , address=79.127.220.101:4048 , piece_id=TVP345FFF2K6UEF2H22BEVWU3O5WH3XDGTKDTUDKYD67OG7O67QA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:33.373868717 (1h4m49.154802094s) [noise=true , address=79.127.205.245:4251 , piece_id=NERWPNU2KEMQZNPDZLSYNNX2J25H4S4HT4ZAPZW652QYMQHJ2WBQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:33.378685049 (1h4m49.159618427s) [noise=true , address=79.127.205.246:4275 , piece_id=ZOLLIB2TPN46WHUMTOWKQL6UGXEC5MESP7KKHCTLOJPAAB5MGT3A, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:34.084339504 (1h4m49.865272884s) [noise=true , address=45.91.116.9:4288 , piece_id=TA7RSWL4GOEOZ3JWXCE62BKOZLMRHLGKN5SGWT4LC3PTZSLXE5NQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:4288): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:35.423681143 (1h4m51.204614521s) [noise=true , address=45.91.116.5:4310 , piece_id=YZ6ULUMBX7YOLR6KVMG5I4RQXGPL3USYJ547GCJEYIWJX6DKEIKA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:35.569580434 (1h4m51.350513818s) [noise=true , address=64.49.13.169:4716 , piece_id=EIOMDZATNT7APLWUNT2VBA76EUNPT477USSBDDT7HECUQRD6BPMQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:35.817104985 (1h4m51.598038363s) [noise=true , address=156.146.43.225:4448 , piece_id=DYKB22KF4QIW3RDZJCDZMXMZ3TV6DPYZFPTUBBNJ3H6GXD3XA66A, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:35.855907345 (1h4m51.636840735s) [noise=true , address=79.127.219.51:4210 , piece_id=E3O7WN4MJXAPYTJ3JBS2QB5DPIEFN2SBPKSWZQDXLP2DNY4MDLDA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:38.925099113 (1h4m54.706032491s) [noise=true , address=64.49.13.170:4719 , piece_id=LCPCJJ7AYZKDGP5JAY4MLJEJO6ZJJHXXLD3Q3KTSSIEMLDLHUARA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:40.437355674 (1h4m56.218289047s) [noise=true , address=45.91.116.11:4151 , piece_id=5MV24DI4E3PR4SFDP5RIK4L3OLBZQC635TU3QHPWDLU3JAKO3XNA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:54.504735029 (1h5m10.28566842s) [noise=true , address=45.91.116.14:4222 , piece_id=BOBNQUQNXQCJRZWZAPQ5GCLQO73Y4437QM7JY66RIU76TBV6ZCNA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 22:57:20.141554727 (1h5m35.922488104s) [noise=true , address=79.127.201.215:4345 , piece_id=QBREW3FB467WCTFL5MB6DPWQXHZVUGBE2PV2LQYZU47AO7B3UTPA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:57:26.897069539 (1h5m42.67800291s) [noise=true , address=79.127.201.215:4350 , piece_id=6K564YWYL4DCFPKMIDIOI7ND5OPD2EJCDTMIBNLBHZQXDWIRW2PQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:57:32.688562247 (1h5m48.469495608s) [noise=true , address=77.90.31.17:4782 , piece_id=5XFMV3EQHUWELC6M7MZQHA7ETAJI3EI6FOZGJ4M5D3DGZ2YV7LFQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk, address:77.90.31.17:4782): protocol: write tcp 159.223.143.39:43290->77.90.31.17:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:57:32.688601675 (1h5m48.469535037s) [noise=true , address=77.90.31.16:4746 , piece_id=WMKPRDVL2SEZJ3UJEQHBIH2JS2JY66ZNUH2XGFRZQDCM2T6OYHWQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: read tcp 159.223.143.39:35968->77.90.31.16:4746: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:57:32.688620974 (1h5m48.469554349s) [seg_pos={0 0}, upload=1003]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 22:57:32.714574471 (1h5m48.49550784s) [upload=1003]: Done uploading segments... Jul 26 22:57:32.714611929 (1h5m48.495545305s) [upload=1003]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration:* 5.782487017s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X......X.............X................X....................] duration: 464.375623ms FAIL (2025-07-26 22:57:37 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 464.410974ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX] duration: 1m0.028842718s FAIL (2025-07-26 22:57:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== .505442529s) [upload=1003]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:56:32.761319638 (1h4m48.542253027s) [seg_pos={0 0}, upload=1003]: Begin upload segment... Jul 26 22:56:32.761432974 (1h4m48.542366355s) [seg_pos={0 0}, upload=1003]: Done begin upload segment. Jul 26 22:56:32.761477820 (1h4m48.542411202s) [noise=true , address=79.127.220.101:4048 , piece_id=TVP345FFF2K6UEF2H22BEVWU3O5WH3XDGTKDTUDKYD67OG7O67QA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.762109645 (1h4m48.543043034s) [noise=true , address=77.90.31.16:4746 , piece_id=WMKPRDVL2SEZJ3UJEQHBIH2JS2JY66ZNUH2XGFRZQDCM2T6OYHWQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.762361611 (1h4m48.543294999s) [noise=true , address=64.49.13.170:4719 , piece_id=LCPCJJ7AYZKDGP5JAY4MLJEJO6ZJJHXXLD3Q3KTSSIEMLDLHUARA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.762780632 (1h4m48.543714033s) [noise=true , address=77.90.31.17:4782 , piece_id=5XFMV3EQHUWELC6M7MZQHA7ETAJI3EI6FOZGJ4M5D3DGZ2YV7LFQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.762908017 (1h4m48.543841405s) [noise=true , address=79.127.201.215:4350 , piece_id=6K564YWYL4DCFPKMIDIOI7ND5OPD2EJCDTMIBNLBHZQXDWIRW2PQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.763409578 (1h4m48.544342966s) [noise=true , address=64.49.13.169:4716 , piece_id=EIOMDZATNT7APLWUNT2VBA76EUNPT477USSBDDT7HECUQRD6BPMQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.763531573 (1h4m48.544464957s) [noise=true , address=79.127.205.246:4275 , piece_id=ZOLLIB2TPN46WHUMTOWKQL6UGXEC5MESP7KKHCTLOJPAAB5MGT3A, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.763877060 (1h4m48.544810515s) [noise=true , address=156.146.43.225:4448 , piece_id=DYKB22KF4QIW3RDZJCDZMXMZ3TV6DPYZFPTUBBNJ3H6GXD3XA66A, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.764091544 (1h4m48.545024933s) [noise=true , address=45.91.116.9:4288 , piece_id=TA7RSWL4GOEOZ3JWXCE62BKOZLMRHLGKN5SGWT4LC3PTZSLXE5NQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.764412768 (1h4m48.545346156s) [noise=true , address=79.127.201.215:4345 , piece_id=QBREW3FB467WCTFL5MB6DPWQXHZVUGBE2PV2LQYZU47AO7B3UTPA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.764678280 (1h4m48.54561167s) [noise=true , address=45.91.116.14:4222 , piece_id=BOBNQUQNXQCJRZWZAPQ5GCLQO73Y4437QM7JY66RIU76TBV6ZCNA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.764860900 (1h4m48.545794285s) [noise=true , address=45.91.116.9:4277 , piece_id=ZRAUWEPI7RWSZ27LW4IEEGLCWQPVPKBUIY24DUDUTYE6VSJUA24Q, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.765166007 (1h4m48.546099397s) [noise=true , address=77.90.31.10:4540 , piece_id=ZP3GZOMNTLQ23WNDVUTDRXCLWZ6WTJLBEVBA4C7OW5N37KCFHKDA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.765283087 (1h4m48.546216475s) [noise=true , address=79.127.205.245:4251 , piece_id=NERWPNU2KEMQZNPDZLSYNNX2J25H4S4HT4ZAPZW652QYMQHJ2WBQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.765695313 (1h4m48.546628702s) [noise=true , address=79.127.219.51:4210 , piece_id=E3O7WN4MJXAPYTJ3JBS2QB5DPIEFN2SBPKSWZQDXLP2DNY4MDLDA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.766142042 (1h4m48.547075431s) [noise=true , address=45.91.116.5:4310 , piece_id=YZ6ULUMBX7YOLR6KVMG5I4RQXGPL3USYJ547GCJEYIWJX6DKEIKA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.766570851 (1h4m48.54750424s) [noise=true , address=77.90.31.11:4581 , piece_id=6TPSO2XZ7GPQ46ZMIEUXITWLMQAMYNOYDB3XWUEX6LLUREBCUULA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.766997991 (1h4m48.547931379s) [noise=true , address=45.91.116.11:4151 , piece_id=5MV24DI4E3PR4SFDP5RIK4L3OLBZQC635TU3QHPWDLU3JAKO3XNA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.812919497 (1h4m48.593852878s) [noise=true , address=77.90.31.10:4540 , piece_id=ZP3GZOMNTLQ23WNDVUTDRXCLWZ6WTJLBEVBA4C7OW5N37KCFHKDA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:32.823436329 (1h4m48.604369716s) [noise=true , address=45.91.116.9:4277 , piece_id=ZRAUWEPI7RWSZ27LW4IEEGLCWQPVPKBUIY24DUDUTYE6VSJUA24Q, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:32.855001154 (1h4m48.635934542s) [upload=1003]: Next returned nil segment Jul 26 22:56:32.855041745 (1h4m48.635975129s) [upload=1003]: Waiting for error group managing segments... Jul 26 22:56:32.915064007 (1h4m48.695997397s) [noise=true , address=77.90.31.11:4581 , piece_id=6TPSO2XZ7GPQ46ZMIEUXITWLMQAMYNOYDB3XWUEX6LLUREBCUULA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:32.975276655 (1h4m48.756210041s) [noise=true , address=79.127.220.101:4048 , piece_id=TVP345FFF2K6UEF2H22BEVWU3O5WH3XDGTKDTUDKYD67OG7O67QA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:33.373868717 (1h4m49.154802094s) [noise=true , address=79.127.205.245:4251 , piece_id=NERWPNU2KEMQZNPDZLSYNNX2J25H4S4HT4ZAPZW652QYMQHJ2WBQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:33.378685049 (1h4m49.159618427s) [noise=true , address=79.127.205.246:4275 , piece_id=ZOLLIB2TPN46WHUMTOWKQL6UGXEC5MESP7KKHCTLOJPAAB5MGT3A, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:34.084339504 (1h4m49.865272884s) [noise=true , address=45.91.116.9:4288 , piece_id=TA7RSWL4GOEOZ3JWXCE62BKOZLMRHLGKN5SGWT4LC3PTZSLXE5NQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:4288): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:35.423681143 (1h4m51.204614521s) [noise=true , address=45.91.116.5:4310 , piece_id=YZ6ULUMBX7YOLR6KVMG5I4RQXGPL3USYJ547GCJEYIWJX6DKEIKA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:35.569580434 (1h4m51.350513818s) [noise=true , address=64.49.13.169:4716 , piece_id=EIOMDZATNT7APLWUNT2VBA76EUNPT477USSBDDT7HECUQRD6BPMQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:35.817104985 (1h4m51.598038363s) [noise=true , address=156.146.43.225:4448 , piece_id=DYKB22KF4QIW3RDZJCDZMXMZ3TV6DPYZFPTUBBNJ3H6GXD3XA66A, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:35.855907345 (1h4m51.636840735s) [noise=true , address=79.127.219.51:4210 , piece_id=E3O7WN4MJXAPYTJ3JBS2QB5DPIEFN2SBPKSWZQDXLP2DNY4MDLDA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:38.925099113 (1h4m54.706032491s) [noise=true , address=64.49.13.170:4719 , piece_id=LCPCJJ7AYZKDGP5JAY4MLJEJO6ZJJHXXLD3Q3KTSSIEMLDLHUARA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:40.437355674 (1h4m56.218289047s) [noise=true , address=45.91.116.11:4151 , piece_id=5MV24DI4E3PR4SFDP5RIK4L3OLBZQC635TU3QHPWDLU3JAKO3XNA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:54.504735029 (1h5m10.28566842s) [noise=true , address=45.91.116.14:4222 , piece_id=BOBNQUQNXQCJRZWZAPQ5GCLQO73Y4437QM7JY66RIU76TBV6ZCNA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 22:57:20.141554727 (1h5m35.922488104s) [noise=true , address=79.127.201.215:4345 , piece_id=QBREW3FB467WCTFL5MB6DPWQXHZVUGBE2PV2LQYZU47AO7B3UTPA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:57:26.897069539 (1h5m42.67800291s) [noise=true , address=79.127.201.215:4350 , piece_id=6K564YWYL4DCFPKMIDIOI7ND5OPD2EJCDTMIBNLBHZQXDWIRW2PQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:57:32.688562247 (1h5m48.469495608s) [noise=true , address=77.90.31.17:4782 , piece_id=5XFMV3EQHUWELC6M7MZQHA7ETAJI3EI6FOZGJ4M5D3DGZ2YV7LFQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk, address:77.90.31.17:4782): protocol: write tcp 159.223.143.39:43290->77.90.31.17:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:57:32.688601675 (1h5m48.469535037s) [noise=true , address=77.90.31.16:4746 , piece_id=WMKPRDVL2SEZJ3UJEQHBIH2JS2JY66ZNUH2XGFRZQDCM2T6OYHWQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: read tcp 159.223.143.39:35968->77.90.31.16:4746: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:57:32.688620974 (1h5m48.469554349s) [seg_pos={0 0}, upload=1003]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 22:57:32.714574471 (1h5m48.49550784s) [upload=1003]: Done uploading segments... Jul 26 22:57:32.714611929 (1h5m48.495545305s) [upload=1003]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration:* 1m0.028865819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X....X........X.........................................] duration: 577.368757ms FAIL (2025-07-26 22:57:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 577.401149ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X....X....X........X........................................] duration: 525.07917ms FAIL (2025-07-26 22:57:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 525.11794ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X......X.............X................X...................] duration: 418.777523ms FAIL (2025-07-26 22:56:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 418.810212ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 228.648159ms FAIL (2025-07-26 22:56:45 UTC) orbiter warning: uplink: too many requests Duration: 228.669936ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.......X........X.X.X...XX.................X.X..........XX..] duration: 6.612994275s FAIL (2025-07-26 22:56:40 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.613103945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXX] duration: 1m0.03039379s FAIL (2025-07-26 22:56:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== NA, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:55:32.674243598 (1h3m48.455176986s) [noise=true , address=79.127.205.246:4286 , piece_id=OQA4AIYP5YZ3F477SPF6GNBMM7SHV5U3KATHY6QQJ3WIBP7CO5EA, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:55:32.677985321 (1h3m48.458918697s) [noise=true , address=77.90.31.21:4907 , piece_id=PXE3CKSU44PJHR5UK5LYIH6VB6HGMLMF635TEXWNZJCSHPLJKN4A, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:55:32.723216630 (1h3m48.504150019s) [noise=true , address=77.90.31.16:4746 , piece_id=I5VVF4LOTMJACG24R6NCQFJITFOGPVQ6DHPHDKNLFEWGZJJIXFOQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:32.771831145 (1h3m48.55276453s) [upload=985]: Next returned nil segment Jul 26 22:55:32.771853345 (1h3m48.552786728s) [upload=985]: Waiting for error group managing segments... Jul 26 22:55:32.807532859 (1h3m48.588466237s) [noise=true , address=77.90.31.21:4907 , piece_id=PXE3CKSU44PJHR5UK5LYIH6VB6HGMLMF635TEXWNZJCSHPLJKN4A, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:55:32.880403589 (1h3m48.66133698s) [noise=true , address=79.127.219.47:4061 , piece_id=6OXKEMPPAQDVFFVMVW7CXA7KRJVYCIZZEHXUOSC7V5KRTUZIQRWQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:33.255832809 (1h3m49.036766172s) [noise=true , address=79.127.205.246:4286 , piece_id=OQA4AIYP5YZ3F477SPF6GNBMM7SHV5U3KATHY6QQJ3WIBP7CO5EA, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:55:33.263794266 (1h3m49.04472763s) [noise=true , address=79.127.205.235:4684 , piece_id=HS7OT27GKKLBDVERCH2D5JHO3SC5SKEITMV3YGFCCMCSWLDI7WYQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:55:33.281905164 (1h3m49.062838532s) [noise=true , address=79.127.205.233:4654 , piece_id=H653RFI273YSI5OZMHNDNRPHZRROJYHQP5QQQVND2DFGQ6OWYQZA, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:55:33.302210041 (1h3m49.083143411s) [noise=true , address=79.127.205.233:4633 , piece_id=X5KEHXPXQ5ZHSBWZRI4QKRUQAKEWDI3E4DTIZ7KQRQGCO7YSLENA, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:55:33.450651005 (1h3m49.231584396s) [noise=true , address=64.49.13.140:4514 , piece_id=3JKSLMMZAQIGXDR4KGKFUHKCDOKM4JOZZXWM2M2MDUQJNFDGTODQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:33.835499140 (1h3m49.616432517s) [noise=true , address=64.49.13.140:4508 , piece_id=UY7JZETDWQBMAD3IC5KSAKTZM2BJZQJNLHGC4JA64FSSH2IC2TNQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:34.505250520 (1h3m50.286183888s) [noise=true , address=64.49.13.138:4483 , piece_id=HWXDZN4M4MAEN46SABLZTZ2GT2PT4LLGYYWLWW64OBULBITAL54Q, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:36.438194749 (1h3m52.219128113s) [noise=true , address=79.127.220.101:4048 , piece_id=HMFY7FMMOOEKR26BYPGMTMCGK2BIRVVILNX54B6DGVEWYL66LWQQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:55:38.274315868 (1h3m54.055249252s) [noise=true , address=45.91.116.10:4100 , piece_id=QNYKZFKTJODQWMQUCTEYTKDDZXTB7F7UB7K5HM7MWB4HOS77ZEVA, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, address:45.91.116.10:4100): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:40.170802737 (1h3m55.951736119s) [noise=true , address=45.91.116.14:4206 , piece_id=7O3U4KDLNN5TBQBCVAVA2ER2TFICPV3CKVBVWLLLWVDRIJJVKKWQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:43.140523230 (1h3m58.921456601s) [noise=true , address=64.49.13.170:4721 , piece_id=LVXHL2NSU2OR53P4GALQMQUGXNRDKIPMMBEE2YI4HIYMOFOR66CQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:55:49.679945213 (1h4m5.460878582s) [noise=true , address=45.91.116.11:4151 , piece_id=7SAM6RVHGM4DFEHMKPPSQG7PZ3F5FSMA4DV2U5OPOMMY3HYVCRPQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:55:59.189919044 (1h4m14.970852414s) [noise=true , address=45.91.116.11:4121 , piece_id=JXUIGBMQXDYD2BTPDHSAFWSNTK56UQXDKKDTSM2SQLMNMPR3YG7Q, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:56:04.555829348 (1h4m20.336762749s) [noise=true , address=79.127.201.215:4356 , piece_id=LBYZQI5UZGPQ5JCY4SY4CEFQB65VPLJVAOXKGJ3NYFJJJ3Q2YMBQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:56:09.533009245 (1h4m25.313942612s) [noise=true , address=79.127.201.215:4344 , piece_id=LFZ4KAK6673NULXX7GW2SAM7X5S52OLYYTLJ6TV5CS4O3K4Y375A, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:56:09.553183918 (1h4m25.334117304s) [noise=true , address=79.127.219.48:4090 , piece_id=H4MW7BEMACRS5JNOD3XCJNOQGPOI6IM2RC245L2VCU5JB5RY4FYQ, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:56:09.553384364 (1h4m25.334317744s) [noise=true , address=79.127.219.47:4061 , piece_id=ZQYG3BYQYAPVTCFEHL7EQUYNT5SRY42MLBCAR3RNJQBXVUCAAP2A, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:56:09.553547502 (1h4m25.334480872s) [noise=true , address=77.90.31.16:4754 , piece_id=XKHYQ5BIWB6XWGP2WYQUHIO445Z2JA5ZW6UXCNXMUBHSK263IVOA, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:56:09.556202697 (1h4m25.337136078s) [noise=true , address=79.127.220.101:4048 , piece_id=TZIJSEQEPNAPVOAW23HVSVEXKSB63USB7CXIZLKI4M6PKZJXYIZA, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:56:09.558563432 (1h4m25.339496809s) [noise=true , address=79.127.219.51:4212 , piece_id=WJNITFJHCAOEV2JL4ZFOO5LAJWXA5RXLOUDKCBJPKTFB7QWGYGJQ, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:56:09.561012107 (1h4m25.341945483s) [noise=true , address=79.127.220.97:4361 , piece_id=X6RK4O5Z7DAUSCWKGM32PPAAHDUKLULIEBHSHHIL5HEB4GTUNMNQ, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:56:09.563700684 (1h4m25.344634049s) [noise=true , address=79.127.201.215:4357 , piece_id=7WTEARAO64KCKVWO7RAN6IWMIV4XKYKFHMSCTYHWZTTIRCJHVUSQ, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:56:09.568467582 (1h4m25.349400958s) [noise=true , address=79.127.219.47:4061 , piece_id=ZQYG3BYQYAPVTCFEHL7EQUYNT5SRY42MLBCAR3RNJQBXVUCAAP2A, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:09.659522614 (1h4m25.440455981s) [noise=true , address=79.127.220.97:4361 , piece_id=X6RK4O5Z7DAUSCWKGM32PPAAHDUKLULIEBHSHHIL5HEB4GTUNMNQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:09.694937995 (1h4m25.475871403s) [noise=true , address=79.127.201.215:4357 , piece_id=7WTEARAO64KCKVWO7RAN6IWMIV4XKYKFHMSCTYHWZTTIRCJHVUSQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:1yXKsyPEVG4W52L7SUmsDGt4oD2D9YyqsfwDVWpjMnMnyVo2Da, address:79.127.201.215:4357): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:09.700602365 (1h4m25.481535748s) [noise=true , address=79.127.220.101:4048 , piece_id=TZIJSEQEPNAPVOAW23HVSVEXKSB63USB7CXIZLKI4M6PKZJXYIZA, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: expected piece hash; storage node overloaded, request limit: 2000; EOF Jul 26 22:56:09.974242450 (1h4m25.755175837s) [noise=true , address=79.127.219.51:4212 , piece_id=WJNITFJHCAOEV2JL4ZFOO5LAJWXA5RXLOUDKCBJPKTFB7QWGYGJQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:10.430438521 (1h4m26.211371893s) [noise=true , address=79.127.219.48:4090 , piece_id=H4MW7BEMACRS5JNOD3XCJNOQGPOI6IM2RC245L2VCU5JB5RY4FYQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:56:32.588868610 (1h4m48.369802037s) [noise=true , address=77.90.31.16:4754 , piece_id=XKHYQ5BIWB6XWGP2WYQUHIO445Z2JA5ZW6UXCNXMUBHSK263IVOA, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: read tcp 159.223.143.39:57244->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:56:32.588907708 (1h4m48.369841096s) [seg_pos={0 0}, upload=985]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 22:56:32.615742699 (1h4m48.396676069s) [upload=985]: Done uploading segments... Jul 26 22:56:32.615778212 (1h4m48.396711598s) [upload=985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.030419854s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X....X....X........X.......................................] duration: 467.482785ms FAIL (2025-07-26 22:56:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.559832ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..X......X.............X................X..................] duration: 477.797089ms FAIL (2025-07-26 22:56:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 477.82976ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 195.180727ms FAIL (2025-07-26 22:56:18 UTC) orbiter warning: uplink: too many requests Duration: 195.273008ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [........X........X.X.X...XX.................X.X..........XX.] duration: 9.708611643s FAIL (2025-07-26 22:56:17 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.708629063s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.........X........X.X.X...XX.................X.X..........XX] duration: 1m0.00062972s FAIL (2025-07-26 22:56:07 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000642837s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..X......X.............X................X.................] duration: 766.413801ms FAIL (2025-07-26 22:56:04 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 766.457054ms core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)