Orbiter History

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 31.431837ms target: https://us1.storj.io/login FAIL (2025-08-08 01:37:37 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 31.474901ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 194.038055ms FAIL (2025-08-08 01:37:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 194.093601ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 558.98537ms FAIL (2025-08-08 01:37:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 559.012522ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 33.502605ms target: https://us1.storj.io/login FAIL (2025-08-08 01:36:53 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 33.521053ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 31.153243ms target: https://us1.storj.io/login FAIL (2025-08-08 01:36:37 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 31.180437ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 963.272704ms FAIL (2025-08-08 01:36:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 963.302768ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 556.715099ms FAIL (2025-08-08 01:36:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 556.744873ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 36.401325ms target: https://us1.storj.io/login FAIL (2025-08-08 01:36:08 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 36.423925ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 982.764391ms FAIL (2025-08-08 01:35:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 982.803441ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 9.199015ms target: https://us1.storj.io/login FAIL (2025-08-08 01:35:43 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 9.279401ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 263.708183ms FAIL (2025-08-08 01:35:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 263.741225ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 710.421814ms FAIL (2025-08-08 01:35:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 710.448993ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 225.299112ms FAIL (2025-08-08 01:35:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 225.343535ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 522.070229ms FAIL (2025-08-08 01:34:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 522.103738ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 196.16149ms FAIL (2025-08-08 01:34:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 196.189481ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 671.341273ms FAIL (2025-08-08 01:34:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 671.372226ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 233.816979ms FAIL (2025-08-08 01:33:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 233.846394ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 539.577505ms FAIL (2025-08-08 01:33:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 539.606749ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 572.595722ms FAIL (2025-08-08 01:32:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 572.623797ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 196.286283ms FAIL (2025-08-08 01:32:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 196.313704ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 455.095125ms FAIL (2025-08-08 01:32:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 455.133897ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 214.228566ms FAIL (2025-08-08 01:31:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 214.252436ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 466.980048ms FAIL (2025-08-08 01:31:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 467.01053ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 584.482465ms FAIL (2025-08-08 01:30:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 584.512466ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 201.6778ms FAIL (2025-08-08 01:30:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 201.707191ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 180.851324ms FAIL (2025-08-08 01:29:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 180.881154ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 764.774678ms FAIL (2025-08-08 01:29:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 764.803637ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 725.692871ms FAIL (2025-08-08 01:29:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 725.722876ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 190.528545ms FAIL (2025-08-08 01:28:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 190.56133ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 527.768896ms FAIL (2025-08-08 01:28:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 527.808494ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.235233353s FAIL (2025-08-08 01:28:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 1.235255548s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 400.463515ms FAIL (2025-08-08 01:27:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 400.502999ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 463.633546ms FAIL (2025-08-08 01:27:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 463.67292ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 176.473239ms FAIL (2025-08-08 01:27:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 176.498363ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 433.662134ms FAIL (2025-08-08 01:26:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 433.692028ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 197.70207ms FAIL (2025-08-08 01:26:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 197.735836ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 433.301467ms FAIL (2025-08-08 01:25:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 433.331428ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 392.881035ms FAIL (2025-08-08 01:25:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 392.917121ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 475.026636ms FAIL (2025-08-08 01:25:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 475.057795ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 204.036201ms FAIL (2025-08-08 01:24:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 204.071724ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 559.072956ms FAIL (2025-08-08 01:23:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 559.112268ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 190.877097ms FAIL (2025-08-08 01:23:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 190.904269ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 965.195066ms FAIL (2025-08-08 01:23:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 965.227348ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 284.595953ms FAIL (2025-08-08 01:23:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 284.70558ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 476.172163ms FAIL (2025-08-08 01:22:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 476.226265ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 231.154226ms FAIL (2025-08-08 01:22:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 231.179305ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 558.196148ms FAIL (2025-08-08 01:22:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 558.221271ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 212.372576ms FAIL (2025-08-08 01:22:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 212.399041ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 38.096654ms FAIL (2025-08-08 01:22:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ... Aug 8 01:13:32.016665378 (15h34m6.531767945s) [upload=16034]: Upload finished. err: <nil> Aug 8 01:13:56.018236159 (15h34m30.533338752s) [upload=16043]: Starting upload Aug 8 01:13:56.018304851 (15h34m30.533407433s) [upload=16043]: Done starting upload Aug 8 01:13:56.018361532 (15h34m30.533464101s) [upload=16043]: Uploading segments... Aug 8 01:13:56.018437882 (15h34m30.533540462s) [upload=16043]: Got next segment. Inline: true Aug 8 01:13:56.018444381 (15h34m30.533546959s) [upload=16043]: Waiting for error group managing segments... Aug 8 01:13:56.018490460 (15h34m30.533593037s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:13:56.018495080 (15h34m30.533597657s) [upload=16043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:13:56.018498109 (15h34m30.533600699s) [upload=16043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:13:56.098636745 (15h34m30.613739343s) [upload=16043]: Done uploading segments... Aug 8 01:13:56.098655758 (15h34m30.613758326s) [upload=16043]: Upload finished. err: <nil> Aug 8 01:14:24.615408874 (15h34m59.130511451s) [upload=16054]: Starting upload Aug 8 01:14:24.615460932 (15h34m59.130563508s) [upload=16054]: Done starting upload Aug 8 01:14:24.615534424 (15h34m59.130637001s) [upload=16054]: Uploading segments... Aug 8 01:14:24.615583186 (15h34m59.130685777s) [upload=16054]: Got next segment. Inline: true Aug 8 01:14:24.615589195 (15h34m59.130691772s) [upload=16054]: Waiting for error group managing segments... Aug 8 01:14:24.615647516 (15h34m59.130750093s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:14:24.615653960 (15h34m59.130756539s) [upload=16054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:14:24.615658359 (15h34m59.130760936s) [upload=16054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:14:24.702851246 (15h34m59.217953823s) [upload=16054]: Done uploading segments... Aug 8 01:14:24.702888715 (15h34m59.217991291s) [upload=16054]: Upload finished. err: <nil> Aug 8 01:15:38.321095418 (15h36m12.836198004s) [upload=16073]: Starting upload Aug 8 01:15:38.321153735 (15h36m12.836256311s) [upload=16073]: Done starting upload Aug 8 01:15:38.321212948 (15h36m12.836315513s) [upload=16073]: Uploading segments... Aug 8 01:15:38.321249839 (15h36m12.836352422s) [upload=16073]: Got next segment. Inline: true Aug 8 01:15:38.321254424 (15h36m12.83635699s) [upload=16073]: Waiting for error group managing segments... Aug 8 01:15:38.321295982 (15h36m12.836398559s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:15:38.321300893 (15h36m12.83640346s) [upload=16073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:15:38.321304075 (15h36m12.836406642s) [upload=16073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:15:38.390829923 (15h36m12.905932488s) [upload=16073]: Done uploading segments... Aug 8 01:15:38.390848766 (15h36m12.905951332s) [upload=16073]: Upload finished. err: <nil> Aug 8 01:16:51.390944653 (15h37m25.906047236s) [upload=16096]: Starting upload Aug 8 01:16:51.391011766 (15h37m25.906114353s) [upload=16096]: Done starting upload Aug 8 01:16:51.391145544 (15h37m25.906248108s) [upload=16096]: Uploading segments... Aug 8 01:16:51.391197811 (15h37m25.906300394s) [upload=16096]: Got next segment. Inline: true Aug 8 01:16:51.391204050 (15h37m25.906306642s) [upload=16096]: Waiting for error group managing segments... Aug 8 01:16:51.391256651 (15h37m25.906359231s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:16:51.391261684 (15h37m25.906364267s) [upload=16096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:16:51.391264863 (15h37m25.906367444s) [upload=16096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:16:51.472233995 (15h37m25.987336571s) [upload=16096]: Done uploading segments... Aug 8 01:16:51.472252641 (15h37m25.987355216s) [upload=16096]: Upload finished. err: <nil> Aug 8 01:18:05.014247695 (15h38m39.529350263s) [upload=16114]: Starting upload Aug 8 01:18:05.014303740 (15h38m39.529406315s) [upload=16114]: Done starting upload Aug 8 01:18:05.014370865 (15h38m39.529473438s) [upload=16114]: Uploading segments... Aug 8 01:18:05.014467752 (15h38m39.529570337s) [upload=16114]: Got next segment. Inline: true Aug 8 01:18:05.014476834 (15h38m39.529579424s) [upload=16114]: Waiting for error group managing segments... Aug 8 01:18:05.014552206 (15h38m39.529654786s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:05.014575699 (15h38m39.529678284s) [upload=16114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:05.014580851 (15h38m39.529683429s) [upload=16114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:05.173375558 (15h38m39.688478134s) [upload=16114]: Done uploading segments... Aug 8 01:18:05.173392883 (15h38m39.68849546s) [upload=16114]: Upload finished. err: <nil> Aug 8 01:18:52.130649287 (15h39m26.645751864s) [upload=16126]: Starting upload Aug 8 01:18:52.130725876 (15h39m26.645828445s) [upload=16126]: Done starting upload Aug 8 01:18:52.130789702 (15h39m26.645892273s) [upload=16126]: Uploading segments... Aug 8 01:18:52.130844581 (15h39m26.645947167s) [upload=16126]: Got next segment. Inline: true Aug 8 01:18:52.130851731 (15h39m26.645954312s) [upload=16126]: Waiting for error group managing segments... Aug 8 01:18:52.130904584 (15h39m26.64600716s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:18:52.130909120 (15h39m26.646011698s) [upload=16126]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:18:52.130912741 (15h39m26.646015319s) [upload=16126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:18:52.217898747 (15h39m26.733001324s) [upload=16126]: Done uploading segments... Aug 8 01:18:52.217935549 (15h39m26.733038116s) [upload=16126]: Upload finished. err: <nil> Aug 8 01:19:55.517120496 (15h40m30.032223072s) [upload=16145]: Starting upload Aug 8 01:19:55.517180374 (15h40m30.032282951s) [upload=16145]: Done starting upload Aug 8 01:19:55.517245083 (15h40m30.03234765s) [upload=16145]: Uploading segments... Aug 8 01:19:55.517315966 (15h40m30.032418533s) [upload=16145]: Got next segment. Inline: true Aug 8 01:19:55.517325222 (15h40m30.032427789s) [upload=16145]: Waiting for error group managing segments... Aug 8 01:19:55.517381429 (15h40m30.032484008s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:19:55.517386048 (15h40m30.032488627s) [upload=16145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:19:55.517389320 (15h40m30.032491899s) [upload=16145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:19:55.743600413 (15h40m30.258702989s) [upload=16145]: Done uploading segments... Aug 8 01:19:55.743616927 (15h40m30.258719502s) [upload=16145]: Upload finished. err: <nil> Aug 8 01:20:29.257594160 (15h41m3.772696735s) [upload=16155]: Starting upload Aug 8 01:20:29.257669520 (15h41m3.772772097s) [upload=16155]: Done starting upload Aug 8 01:20:29.257763018 (15h41m3.772865587s) [upload=16155]: Uploading segments... Aug 8 01:20:29.257821897 (15h41m3.772924477s) [upload=16155]: Got next segment. Inline: true Aug 8 01:20:29.257826572 (15h41m3.772929149s) [upload=16155]: Waiting for error group managing segments... Aug 8 01:20:29.257891306 (15h41m3.772993878s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:20:29.257896584 (15h41m3.77299916s) [upload=16155]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:20:29.257900084 (15h41m3.773002662s) [upload=16155]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:20:29.324124067 (15h41m3.839226651s) [upload=16155]: Done uploading segments... Aug 8 01:20:29.324146082 (15h41m3.83924866s) [upload=16155]: Upload finished. err: <nil> Aug 8 01:21:06.298907732 (15h41m40.814010326s) [upload=16166]: Starting upload Aug 8 01:21:06.298993044 (15h41m40.814095611s) [upload=16166]: Done starting upload Aug 8 01:21:06.299105277 (15h41m40.814207854s) [upload=16166]: Uploading segments... Aug 8 01:21:06.299160453 (15h41m40.814263038s) [upload=16166]: Got next segment. Inline: true Aug 8 01:21:06.299167359 (15h41m40.814269942s) [upload=16166]: Waiting for error group managing segments... Aug 8 01:21:06.299225227 (15h41m40.81432781s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:21:06.299231100 (15h41m40.814333685s) [upload=16166]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:21:06.299234372 (15h41m40.814336952s) [upload=16166]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:21:06.367725210 (15h41m40.882827793s) [upload=16166]: Done uploading segments... Aug 8 01:21:06.367746234 (15h41m40.882848808s) [upload=16166]: Upload finished. err: <nil> Aug 8 01:22:05.005652421 (15h42m39.520754988s) [upload=16184]: Starting upload Aug 8 01:22:05.005701629 (15h42m39.520804207s) [upload=16184]: Done starting upload Aug 8 01:22:05.005780317 (15h42m39.520882894s) [upload=16184]: Uploading segments... Aug 8 01:22:05.005831789 (15h42m39.520934371s) [upload=16184]: Got next segment. Inline: true Aug 8 01:22:05.005837134 (15h42m39.520939711s) [upload=16184]: Waiting for error group managing segments... Aug 8 01:22:05.005889408 (15h42m39.520991984s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 01:22:05.005893853 (15h42m39.520996429s) [upload=16184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 01:22:05.005897232 (15h42m39.52099981s) [upload=16184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 01:22:05.013405083 (15h42m39.528507659s) [upload=16184]: Done uploading segments... Aug 8 01:22:05.013421146 (15h42m39.528523713s) [upload=16184]: Upload finished. err: metaclient: Too Many Requests Duration: 38.119729ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 506.167025ms FAIL (2025-08-08 01:21:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 48 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122292523 (15h13m52.6373951s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.122655357 (15h13m52.637757933s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127239504 (15h13m52.642342091s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127688378 (15h13m52.642790961s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.127985843 (15h13m52.643088424s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128258315 (15h13m52.643360891s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128584811 (15h13m52.643687396s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.128946611 (15h13m52.6440492s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129325847 (15h13m52.644428426s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.129642657 (15h13m52.644745251s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130117633 (15h13m52.645220215s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130438929 (15h13m52.645541508s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.130845916 (15h13m52.645948501s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131182211 (15h13m52.646284778s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.131530486 (15h13m52.646633071s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Uploading piece... Aug 8 00:53:18.230607322 (15h13m52.74570989s) [upload=15682]: Next returned nil segment Aug 8 00:53:18.230623159 (15h13m52.745725725s) [upload=15682]: Waiting for error group managing segments... Aug 8 00:53:18.242749045 (15h13m52.757851614s) [noise=true , address=104.36.82.60:4348 , piece_id=ZKLZJRIO723L2COYKZ7VBLW3XUXNKUX6TSI4ANI5PT4PRJDXVPPA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.244576549 (15h13m52.759679116s) [noise=true , address=104.36.82.44:4887 , piece_id=OVQVBZPHP4K5J2NZGQ5GWKXM2Y4PONIHSABXA2RO2LA4NTOZQOIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.251827136 (15h13m52.766929705s) [noise=true , address=104.36.82.23:4456 , piece_id=5YWV7LPEYQRDB4YQMU6IUXUIVTTEUPWTYYVIAHYPNIOOUVDPQY2Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.252451281 (15h13m52.767553856s) [noise=true , address=104.36.82.16:4314 , piece_id=RW2O5DE6EJWUP2Z6IMAL6KUMCCE5Y4L6HZB7RVJIWRBKPQ6R3OSA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.253652377 (15h13m52.768754952s) [noise=true , address=104.36.82.62:4436 , piece_id=SKTKM7TAQ3AUZQKVZ7WF3MU42UUILXWD2PJBAERN2CTMCGGSDAMA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.262606397 (15h13m52.777708985s) [noise=true , address=104.36.82.11:4117 , piece_id=H7NHN6523ULSHPVXABIQXJVMXL3YMPGHXPBLQUFFQ42YRSOAC3NQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.265863723 (15h13m52.780966304s) [noise=true , address=104.36.82.3:4792 , piece_id=2KBSC7QL2NP3HNZ7SZYQKDQOFZL6EFVHQZHJA6DGYM4LDXXEXNRA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.266994624 (15h13m52.78209719s) [noise=true , address=104.36.82.39:4732 , piece_id=V7DFYXEGQWT5DZIWS6XLANWGBGLHWAUZHSTZ45HRU7X7NSR5GWIA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270637696 (15h13m52.785740258s) [noise=true , address=104.36.82.54:4181 , piece_id=6QYOCEAAN56H37T3OPMVSAMGRFJKSYNSV6OF6YIB3TPC2EP3MQTA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.270741606 (15h13m52.78584417s) [noise=true , address=104.36.82.47:4552 , piece_id=ULDPQPGYZUTQXTGA46VFZH5VSYPTWSY4VIXH7FEV6CWXF2OHRQSQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.275223534 (15h13m52.790326108s) [noise=true , address=104.36.82.4:4814 , piece_id=OUTM5XGOWY2BHJM4FWTDCJI277QYUWIPC56GSLQMYHEPC5OCH54Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.287208624 (15h13m52.802311194s) [noise=true , address=104.36.82.8:4002 , piece_id=5QU2DJ43QHOD326EZQHUDSISYP3JSTAMFZQBT2Z5JM6YGXEGHVUA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.298071431 (15h13m52.813174s) [noise=true , address=104.36.82.59:4319 , piece_id=FB7GZQQJYNLXEY23VTMTQLIPIAZENNEII77LFUGPDH3SAWFTLRQA, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355392162 (15h13m52.870494741s) [noise=true , address=104.36.82.54:4188 , piece_id=QYQVT26JBLEB4WX62GA2BNVMKUIZHMMUQ2X5J2QQZXCANMNDMWWQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: <nil> Aug 8 00:53:18.355421695 (15h13m52.870524272s) [seg_pos={0 0}, upload=15682]: Segment reached optimal threshold of 14 pieces. Aug 8 00:53:18.355889099 (15h13m52.870991662s) [noise=true , address=64.49.13.170:4726 , piece_id=IAZG5BX624Q3S5JBBL7DKK7DELJJKOFUTY3ZM27T5IC4QMC76NRQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o5oB3BoEaPzgXKcqLwvPpjNtS9hNFBZ7zgh7EJhegVRjagakC): protocol: read tcp 159.223.143.39:35006->64.49.13.170:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355942822 (15h13m52.87104539s) [noise=true , address=79.127.205.233:4627 , piece_id=YGXK4MII44PCFLZMA5IJ2T4TNYKPPRH5TR4G4ECC7NMEYLJ3EY3Q, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uts4vuBSzwvxtAVPCX2jF7gxEcJrBXGgfNRi3rWM2279NCgm): protocol: read tcp 159.223.143.39:48546->79.127.205.233:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355964966 (15h13m52.871067528s) [noise=true , address=64.49.13.139:4474 , piece_id=O6H7JRJ5ZVKOQEBGC3DDEB2XQJFFXL2B5EKBIQQA23OC3QTTUXIQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126wi7mavkxXLAWpoK1VP6QX2QPddsydzkMNRJc5uDU3LrCRVY2): protocol: read tcp 159.223.143.39:39484->64.49.13.139:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.355991288 (15h13m52.87109385s) [noise=true , address=79.127.205.233:4631 , piece_id=KIA4EYX5MYCV2TV4EQ3WILY36AYC3QRVR55WE5N3DW3R4JR3A5UQ, seg_pos={0 0}, upload=15682]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tNcNixo72dz84o6bJjxoAnDByxLeHwEDBBwNrE3R5MvTCRr4v): protocol: read tcp 159.223.143.39:44010->79.127.205.233:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:53:18.356018268 (15h13m52.871120829s) [seg_pos={0 0}, upload=15682]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:53:18.356076234 (15h13m52.87117881s) [upload=15682]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:53:18.356079897 (15h13m52.871182474s) [upload=15682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:53:18.412919044 (15h13m52.928021629s) [upload=15682]: Done uploading segments... Aug 8 00:53:18.412957531 (15h13m52.928060122s) [upload=15682]: Upload finished. err: <nil> Aug 8 00:53:51.325422002 (15h14m25.84052457s) [upload=15691]: Starting upload Aug 8 00:53:51.325472857 (15h14m25.840575433s) [upload=15691]: Done starting upload Aug 8 00:53:51.325560802 (15h14m25.84066337s) [upload=15691]: Uploading segments... Aug 8 00:53:51.325741160 (15h14m25.84084374s) [upload=15691]: Got next segment. Inline: false Aug 8 00:53:51.325779261 (15h14m25.840881829s) [upload=15691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:53:51.325783783 (15h14m25.840886359s) [upload=15691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:53:51.334218294 (15h14m25.849320873s) [upload=15691]: Next segment err: metaclient: Too Many Requests Aug 8 00:53:51.334246922 (15h14m25.849349485s) [upload=15691]: Done uploading segments... Aug 8 00:53:51.334262232 (15h14m25.849364794s) [upload=15691]: Upload finished. err: metaclient: Too Many Requests Duration: 506.196135ms core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)